Warped Optical-Flow Inter-Frame Reconstruction for Ultrasound Image Enhancement

Problem statement: Optical flow inter-frame enhancement is one of the echniques to improve the quality of ultrasound images by reducin g the speckle noise. The performance depends on accuracy of the optical flow inter-frame reconstruc ion which is a part of the technique. The speckle noise of an ultrasound image forms certain pattern, of which size of the speckle varies depends on the distance from the transducer. Approach: This study proposed that the use of warping techniq ue prior to as well as after the optical flow to improve the accuracy of the optical flow calculation. The warp ing function was derived to transform the image from ci rcular grid to rectangular grid. Results: The technique was applied to a series of synthetic movi ng frames generated from actual ultrasound image taken from a patient heart. The technique improved th accuracy of motion vectors generated from the optical flow. The MSE of the reconstructed frame wa s also smaller using warped technique compared to that of the non-warped technique. Conclusion: The proposed warping operation was shown to be successful in increasing the accuracy of the frame reconstruction for the optical flow inter-frame enhancement technique.


INTRODUCTION
Ultrasound imaging is a very popular modality among various medical imaging techniques due to the low operational and capital costs as well as the low safety requirements (Ngah et al., 2007;Hajialioghlo et al., 2009). However, it has certain drawbacks caused by the nature of the ultrasound waves used for the source. The interference between the ultrasound waves generates speckle noise that degrades the sharpness of the image. Therefore, the image needs to be enhanced prior to utilizing it for medical diagnosis. Many techniques have been developed in order to reduce the speckle noise in ultrasound images, including wavelet filters (Rallabandi, 2008), fuzzy filters (Rafiee et al., 2004), anisotropic diffusion techniques (Li and Meng, 2007) and optical flow inter-frame technique . Most of these techniques were implemented in a rectangular grid, while the granular speckle noise in ultrasound follows direction.  applied anisotropic diffusion technique in a circular grid by warping operation, which increased the PSNR of the filtered image.
One of the speckle noise reduction techniques is optical flow inter-frame technique, which utilizes three consecutive frames extracted from a video produced by an ultrasound imaging device . By reconstructing a new image based on the first and the third frames using optical flow technique, the speckle noise of the second frame can be reduced. This technique is proper for enhancing the image quality of moving objects, e.g. cardiac motion. However, the performance of this technique in reducing speckle noise depends on the accuracy of the optical flow in tracking the movement of the speckle noise in the consecutive images.
It can be seen that the size of the speckle noise is a function of the radial distance from the center of the speckle patterns. Hence, when an object moves from on one position to another position at different radial distances, the speckle noise size will also change. However, when the rectangular grid is transformed into a circular grid, the size of the speckle noise will no longer depend on the radial distance. Hence, the accuracy of the optical flow in tracking the speckle movement will be improved.
The previous optical flow inter-frame technique was applied in a rectangular grid too . This study adds warping operation to transform the image from rectangular to circular grid prior to the optical flow inter-frame technique to increase the accuracy of frame reconstruction for image enhancement.

MATERIALS AND METHODS
Ultrasound speckle noise: Ultrasound imaging or sonography is a non invasive medical imaging which uses ultrasound waves as a source to interrogate the morphology of organs, vessels and tissues inside a patient body. A transducer transmits an ultrasound wave with certain frequency against patient skin. When the ultrasound wave strikes an object, part of it is reflected back. The transducer also acts as a receiver that receives the bounced wave. By recording the time-of-flight of the ultrasound wave, the distance between the transducer and the object can be determined. The transducer is rotated with certain steps to perform scanning to the object. The measured data is then used to reconstruct an image representing the object being scanned. An example of ultrasound image is given in Fig. 1.
It can be seen in Fig. 1 that the image is constructed by many random granules composing an object. The granules are actually speckle noise that generated by the interactions between ultrasound waves. The speckle noise forms a specific pattern which is a circular pattern that centered to the transducer. The sizes of the granules vary according to their distance from the transducer. The farther a granule from the center is, the larger its size is.   It can be seen in Fig. 1 that the granules at position A on the top of the image, which are closer to the transducer, are smaller than the granules at position B on the bottom of the image. Therefore, when the granules move due to the movement of the object being scanned, their sizes also change according to their position in the image. This trend should be considered when we utilize the movement of the granules to analyze or to further process the image as in optical flow techniques.
Warping operations: In order to overcome the variability of the granular size while moving to different radial distance to the transducer, warping operation is introduced. The warping operation is actually transforming the image from a circular grid into a rectangular grid. The circular grid applied to an ultrasound image is depicted in Fig. 2. Using this grid, the direction of the granular speckles will match with the grid. This image is then warped in the way that the circular grid is transformed into a rectangular grid, as shown in Fig. 3. Afterwards, the warped image is ready to be processed using the optical flow inter-frame enhancement technique. The transformation function for the warping operation is derived as illustrated by Figs. 4 and 5. The original image has an x-y coordinate system, with x as the horizontal axis and y as the vertical axis. The warped image has an s-r coordinate system, with s as the horizontal axis and r as the vertical axis. In the original image, r' represents the radial distance from the probe.
An imaginary pie can be drawn over the original image, covering the image generated by the ultrasound imaging device (Fig. 4). The probe is located at the center of the pie, with coordinate (x 0 , y 0 ). R is the radius of the pie w and h is the width and height of the original image θ is an angle measured from the vertical axis of the probe. θ L and θ R are the angles from the vertical axis of the probe to the left edge and right edge of the pie, respectively.
With the warping operation, the pie is bent hence coordinates of points A to H on the original image moved to points A' to H' on the warped image. To map the original image into the warped image, we need to find the relationship between x and y with s and r. As in image processing practices, inverse transformation functions are more preferable than forward transformation functions for geometrical operations. Therefore, we need to calculate x and y as functions of s and r.
Using geometrical proportion principle, the relationship between s and θ can be derived as Eq. 1: The width of the original image is w pixels. However, since the coordinate of the pixel started from 0, the horizontal coordinate of rightmost pixel is w-1. The negative sign of θ L is due to its reverse direction to θ. As previously mentioned, the transformation function will be θ as a function of s instead of s as a function of θ. Hence, Eq. 1 can be rearranged into Eq. 2: Likewise, the relationship between r' and r is as Eq. 3: Also as previously mentioned, the transformation function is r' as a function of r instead of r as a function of r'. Hence, Eq. 3 can be rewritten into Eq. 4: The original coordinates (x and y) can then be calculated using Eq. 5 and 6: After the enhancement process, the resulted image should be warped back to the original coordinate system. In this process, called as dewarping process, the pixel coordinate in rectangular grid are remapped back into circular grid using the following equations. Now, we need to calculate s and r as functions of x and y. Eq. 5 and 6 can be rewritten as follow: Dividing Eq. 7 with 8 gives the value of θ as function of x and y: Eq. (5) can also be rearranged into Eq: (10): Finally, s and r can then be calculated using rearrangements of Eq. (1) and (3): Optical Flow Inter-Frame Enhancement Technique: Optical flow inter-frame enhancement is a technique which uses a series of three frames to reduce the speckle noise . The frames are extracted from a video taken from an ultrasound imaging device. The three images are the input of the technique, while the output is an enhanced version of the second image. An optical flow algorithm is applied to the first and the third images to calculate the movements of objects inside the images. The distances and directions of the movements can then be used to reconstruct a new image, which in turn will be used to enhance the second image.
The main idea of this technique is that for small movement, the movements of objects inside the image are relatively constant between two consecutive frames in terms of velocity and direction. Therefore, if we track an object between three consecutive frames, the position of the object in the second frame is in the middle of its positions in the first frame and third frame, as illustrated in Fig. 6. P 1 (x 1 , y 1 ) is the position of the object in the first frame, while P 2 (x 2 , y 2 ) and P 3 (x 3 , y 3 ) are the positions of the object in the second and third frames, respectively. By calculating the movement from P 1 to P 3 , the movement to P 2 can be determined using Eq. 13: 1 3 1 2 P P P P 2 = From this equation, the coordinate of the second position can be calculated using Eq. 14 and 15 as follow.
The schematic diagram of the technique is given in Fig. 7. The first step of the technique is extraction of three frames from the ultrasound video. The frames are two-dimensional B-mode images acquired from the ultrasound imaging device.
Using an optical flow algorithm, all objects in the first frame are tracked hence their positions in the third frames are known. The algorithm produces motion vectors for all pixels in the first frame. Assuming the position of an object in the second frame is halfway between the first and the third frames, the magnitude of these motion vectors are divided by two to obtain the magnitude of the motion vectors from the first frame to the second frame. The directions of the motion vectors remain the same.
The next step is frame reconstruction. Having obtained the positions of all pixels in the second frame, the values of these pixels are copied from their original locations in the first frame.
I 1 (x 1 , y 1 ) is the intensity of the pixel at coordinate (x 1 , y 1 ) on the first input frame. I 2 (x 2 , y 2 ) is the intensity of the pixel at coordinate (x 2 , y 2 ) on the reconstructed frame, with x 2 and y 2 obtained using Eq. (14) and (15). Since generally the calculated positions of the pixels in the second frame are not integer values; an inverse image spatial transformation operation is applied using bicubic interpolation of the neighboring pixels in the first frame.
The last step of this technique is image fusion, which is basically a logarithmic averaging operation between the original second frame and the frame reconstructed from the optical flow algorithm.  Fig. 7, consists of optical flow inter-frame reconstruction, which is the combination between optical flow and mapping operations. If we focus on this step, the original optical flow inter-frame reconstruction will be given in Fig. 8. In this study, we modify this reconstruction by introducing warping operation prior to the optical flow optical flow inter-frame reconstruction as well as dewarping operation afterward. This modification is illustrated in Fig. 9.
Quantitative evaluation: In order to evaluate the performance of the proposed technique quantitatively, a series of three synthetic frames are generated from an actual ultrasound image. The first frame used in the optical flow inter-frame algorithm is the actual frame taken from a ultrasound video. The second frame is then generated by translating the first frame with certain distance and direction. The third frame is generated by further translating the frame with the same distance and direction. The translation also considers that when a granular speckle noise move from one radial distance to another, the size of the speckle changes. The synthetic frames process is ilustrated by Fig. 10.
Using these frames, the optical flow inter-frame reconstruction technique is applied. The inputs of the optical flow algorithm are the actual first frame and the generated third frame. The generated second frame will be used as a baseline of the reconstructed image, which will be used to determine the accuracy of the optical flow frame reconstruction for inter-frame image enhancement. This is done by calculating the mean square error (MSE) of the reconstructed frame to the target using Eq.
I R (x, y) is the pixel intensity of reconstructed image at coordinate (x, y). I B (x, y) is the pixel intensity of baseline reconstructed image at coordinate (x, y). ROI is the region of interest, which is a part of the image that will be used for the calculation of the MSE. N is the number of pixels on the ROI.
The proposed warped technique will be compared with the original technique by comparing the MSE of both techniques. The technique with lower MSE will be considered as the better one.

RESULTS
The data used to test the proposed technique are obtained from the Cardiology Unit Hospital Universiti Kebangsaan Malaysia Kuala Lumpur. Ultrasound videos of patient hearts were taken and extracted into series of ultrasound frames and inputted to the Synthetic Frame Generation algorithm as well as the Original and Warped Optical-Flow Inter-Frame Reconstruction algorithms.
A frame was extracted from an ultrasound video of a patient heart taken from 4-chamber view. The frame, namely Frame 1, was then translated according to Fig.  10 into Frame 2 and Frame 3. The three frames are given in Fig. 11. Frame 2 ( Fig. 11b) will be used as a ground truth, hence the base for MSE calculation.
All the three synthetic images were then warped by applying Eq. (4-6). The warped images are shown in Fig 12. Frame 1w is the warped version of Frame 1, while Frame 2w and Farame 3w are the warped version of Frame 2 and Frame 3, respectively.
The optical flow inter-frame reconstruction algorithms were then applied to the synthetic frames. The original optical flow inter-frame reconstruction algorithm was applied using the non warped synthetic frames: Frame 1, Frame 2 and Frame 3. Whereas, the warped optical flow inter-frame reconstruction algorithm was applied to the warped synthetic frames: Frame 1w, Frame 2w and Frame 3w. The optical flow algorithm that we used in this study was a wavelet optical flow which was modified from (Saputro et al., 2010). The optical flow generated motion vector that shows the movement of speckle noise. This vector is then superimposed on the first frame and shown in Fig. 13.  Some excursions from the main direction nevertheless exist in several positions within the frames. However, we only interested in the areas where the heart muscles and hence the speckles are located. In order to show the effect of the warping operation prior to the application of optical flow, Fig. 14 zooms portions of the frames enclosed by rectangles in Fig. 13, where the excursions happen. Fig. 14a shows the motion vectors of the non-warped frame, while Fig. 14b shows those of the warped frame.
The motion vectors from the optical flow calculations were then used to reconstruct the second frame using Eq. 14-16. The reconstructed frames for both non-warped and warped frames are shown in Fig.  15, namely Frame 2R and Frame 2wR, respectively. In order to compare to the non-warped reconstructed frame, the warped reconstructed frame was then dewarped back to the original coordinate system using Eq. 7-10, resulting Frame 2dR. Frame 2 is presented again here to ease examining the result.
For quantitative evaluation, the MSE of both nonwarped and warped reconstructed frames were calculated using Eq. 17. The result is presented in Table 1.

DISCUSSION
In the original unwarped frames, the speckle noises have different size as illustrated by Fig.1. After warping operation, the speckle noises in the warped synthetic frames as shown in Fig. 12 now have relatively the same width regardless their positions within the image. The pattern of all the speckles now also become paralel to horizontal.
By observing the motion vector in the non-warped frame (Fig. 13a), the speckle generally moves outward from the center of the speckle pattern where the ultrasound transducer is located. Meanwhile, in the warped frame as shown in Fig. 13b, the movement of the speckle, in general, is uniformly downward vertically as expected.
Obviously by comparing the uniformity of the direction of the motion vector in the non-warped and warped frames (Fig. 14), the speckle motion of the warped frame is not diverted as much as the nonwarped frame. In another word, the optical flow algorithm gives more accurate result in calculating the motion vectors of the speckle on the warped frames compare to the non-warped frames.
Visual observasions were done by matching the reconstructed frames in Fig. 15 with Frame 2. Visually, the dewarped reconstructed frame (Frame 2dR) is closer to the original frame (Frame 2) compare to the non-warped recontructed frame (Frame 2R). It can be seen from Table 1, that the warped technique gives smaller MSE, hence we can conclude that this technique perform better than the original technique.

CONCLUSION
In this study we apply warping operation in the inter-frame optical flow enhancement for ultrasound images by mean of image reconstruction. We observed that the speckle in the b-mode ultrasound image forms certain pattern that varies its size depends on the position of the speckle. The warping operation can uniform the speckle width that hopefully will increase the accuracy of the optical flow. The experiments using hypotethical images show that the optical flow motion vectors of the warped technique is more accurate compare to the non-warped technique. The MSE of the