IMPROVED INTERACTIVE METHOD FOR RECO-VERING 2.5D MODELS FROM SINGLE IMAGES

In the last years the development of interactive Computer-based methods for building virtual and physical 2.5D models from single shaded images faced with an exponential growth. In particular, a wide range of methods based on image processing-based procedures and on Shape From Shading (SFS) can be documented. On the basis of the most favorable techniques devised in literature, the present work describes an improved interactive method capable of retrieving 2.5D models using image shading information. The pro-posed method performs a SFS-based reconstruction where (1) the overall geometry of the expected surface is first recovered and (2) the final 2.5D reconstruction is obtained by minimizing a suitable functional using the rough surface as an initialization function. The method improves previous interactive works by introducing a novel two-step rough surface recovery and a new definition of a functional to be minimized for solving the SFS problem. Tested against a set of case studies the proposed method proves to be effective in providing 2.5D models.


INTRODUCTION
In recent years a strong improvement of Computerbased methods for retrieving shapes from single shaded images can be documented (Remondino and El-Hakim, 2006;Stylianou and Lanitis, 2009;Muruganathan et al., 2014). This is particularly true when dealing with simplified 3D models (Algabary et al., 2014;Vani et al., 2012), such as virtual bas-relief representations (also named 2.5D models). A digital bas-relief delivers a volumetric projection of shapes into the viewer's space, so that it is detached from the two-dimensional background (Kerber et al., 2010); as a result, a nearly flat surface maintaining as much as possible the perception of the full 3D scene can be generated (Weyrich et al., 2007). A major issue in the retrieval of this "nearly 3D shape" is that it results in an ill-posedness of problem as stated by (Tsai and Shah, 1994) consequently, relevant literature works are typically aimed to solve it under certain circumstances, often by means of user interaction. Image embossing (Huang et al., 2011;Golchin et al. 2013) is one of the most used methods for recovering a 2.5D model from single image. It consists of a computer-based method where each pixel, for a given image, is replaced either by a highlight or a shadow, depending on light/dark area boundaries on the original image. The result visually resembles a bas-relief but, due to the algorithm based on image gradient computation, depth is reconstructed in an inconsistent way. Other techniques (e.g., the one proposed by Sun et al., 2009) for improving embossing-based methods have been proposed so far, mainly based on image pre-processing such as histogram equalization and image enhancement.
Also into commercial software packages, like JD Paint (Wang et al., 2010) and Art CAM (www.artcam.com), are developed specific tasks for 2.5D reconstruction. However, these packages are mainly CAD modelers where user interaction is a strongly required in both the pre and post-Science Publications JCS processing phases. Another widely known technique, besides implemented in commercial software, is the socalled "image inflation". This technique consists in constraining the outlines of the object to be virtually reconstructed (Igarashi et al., 2007;Repenning, 2005) and, subsequently, in inflating the surface. A different perspective in 2.5D model reconstruction is provided by Shape From Shading (SFS) techniques, a set of computational approaches whose main aim is to re construct the three-dimensional shape of a surface depicted in a single grey-level image. In detail, as described by (Durou et al., 2008), SFS methods are based on the hypothesis that the image pixel brightness linearly depends on the angle between the surface normal and the direction of scene illumination. Even if SFS proves to be effective in reconstructing shapes of shaded objects, unfortunately they prove to be unsuitable for producing high-quality bas-reliefs (Wang et al., 2010) unless a certain degree of human interaction is provided. To overcome this drawback, a number of interactive methods have been proposed (Daniel and Durou, 2000;Wu et al., 2008); as a matter of fact, interactive SFS methods may be considered among the best candidate techniques for generating high quality 2.5D models starting from single images. For this reason, in a previous authors' paper (Governi et al., 2014) a method combining image filtering and SFS for retrieving 2.5D models starting from single images was proposed. In the above cited approach, authors minimize a suitable functional (i.e., the so called SFS problem) consisting of two terms, namely brightness and smoothness constraints. The minimization is carried out using as initialization function a surface roughly resembling the shape of the image to be reconstructed (i.e., a smoothed version of the original shaded image). This is obtained by applying a Gaussian low-pass filter. Even if the results of this method are quite robust, the use of a Gaussian filter introduces two variables to be set: The size of filter kernel and the value of Gaussian standard deviation. Moreover, the former method requires a manual, optimal, setting of a weight applied on smoothness constraint; the greater is the chosen value, the smoother is the retrieved solution. Accordingly, with the aim of improving the method proposed in (Governi et al., 2014) the present work describes a 2.5D reconstruction method where the smoothed surface used for initializing the SFS problem is obtained by using a proper polynomial approximation of the (discrete) surface built from theimage height map. This allows the definition of a smoothed surface without the need of taking into account filter kernel and standard deviation. Moreover, the 2.5D model retrieval is obtained by minimizing a new, improved and functional where both smoothness and brightness constraint are weighted. As described below, once the initialization function is obtained, the setting of the weights for the two constraints results quite easy.

MATERIALS AND METHODS
The proposed method starts from a given shaded digital image of the object to be reconstructed into a 2.5D representation and consists of the following main steps, as described in Fig. 1

Definition of a Suitable Functional E
The first step consists in the definition of a suitable functional whose minimization allows to recover the expected shape of the object. In this work, such a functional is obtained under the following specific hypothesis, usually adopted in almost all SFS-based techniques: The surface to be reconstructed is completely diffusive (i.e., Lambertian); the albedo is constant in the entire reconstruction domain; the light source is set at infinity; image is free from perspective distortion.
If all the above conditions are fulfilled, it is possible to state the historically known SFS problem (Horn, 1970) where, is the unit-vector opposed to light direction, is the outward unit-length vec-tor normal to the surface (unknown of the problem), is the albedo and is the image brightness; since starting image is grayscale, this term indicates both the image and its brightness. This equation, where the unknown is the vector, is usually expressed using surface gradient as unknown term thus resulting in the following non-linear Partial Differential Equation 2 (PDE): Several works are in literature for solving Equation1, mainly based on direct methods (Kimmel and Bruckstein, 1995;Prados et al., 2006;Rouy and Tourin, 1992), minimization methods (Frankot and Chellappa, 1988;Governi et al., 2013;Horn, 1990;Ikeuchi and Horn, 1981), local approximation methods (Lee and Rosenfeld, 1985) and linear approximation methods (Tsai and Shah, 1994).
Among them, minimization methods are acknowledged to afford the right compromise between efficiency and flexibility leading to robust results also when the starting point consists of a noisy image or imprecise illumination settings (e.g., guessed light direction when unknown). Minimization methods are based on the hypothesis that the expected (reconstructed) surface is the one that minimizes a suitable functional, composed by the sum of several contributions (called "constraints") and usually representing the error between the (iteratively) reconstructed surface and the expected one. In the present work, such a functional is built as a linear combination of brightness and smoothness constraints, as follows Equation 3 to 5: where, and where i is the pixel index, j is the index of a generic pixel belonging to the 4-neighbourhood of ith pixel, I is the brightness of pixel i (range [0-1]), i N and are the unit length vectors normal to the surface (unknown) in positions i and j, 1 λ is a regularization factor for brightness constraint and 1 λ is a regularization factor for smoothness constraint.
Since both of the two constraints -B (I) and S (I)-are quadratic, the resulting functional is a quadratic form too. n n n n n n n n n Φ = (6) where, is the overall number of pixels; as a result, the functional can be rewritten in a matrix form Equation 7: Whose minimization can be carried out by minimizing its gradient Equation 8: where, A(I) (size k×k ) is a sparse symmetric matrix. Depending on values set for regularizing smoothness and brightness constraint, the definition of A(I) values changes considerably i.e., a different functional can be defined for any choice of values 1 λ and 1 λ . The indirect minimization of the functional expressed in Equation 8 allows to provide the final solution of the SFS problem, i.e., the expected surface. The minimization can be accomplished by applying well known linear methods (e.g., Jacobi, Gauss-Seidel, Successive-Over-Relaxation (SOR) etc.) since they provide fast convergence to a minimized (optimized) solution.

High Frequency Details Removal
Depending on the values selected for 1 λ and 1 λ the surface reconstruction may differ in terms of higher details (using higher values for) or higher smoothness (using higher values for 1 λ ). A correct balancing between the two regularizing factors could allow a reliable reconstruction by taking into account both contributions.

JCS
However, unrelatedly of the method used for minimizing Equation 8, one of the main drawbacks of direct minimization is that it often falls to the nearest local minimum (instead of the global ones). Consequently, the optimal setting of regularizing factors is not sufficient to guarantee a correct reconstruction.
In order to overcome this shortfall, minimization algorithms need to be suitably initialized by imposing an initial guess to the solution. A convenient way to provide a reliable initial guess of the final solution (i.e., to initialize properly the minimization procedure) consists of creating a low-frequency version of the desired surface. A known method for discarding image high-frequency details is to use a convenient low-pass filter; by way of example, in (Governi et al., 2014) a Gaussian filter is used as a main step for retrieving the rough surface. However, smoothing filters are kernel-based operators and, therefore, their effectiveness depends on kernel dimension and neighbor pixel values. With the aim of retrieving a rough surface globally, i.e., regardless of the local pixel values and the kernel size, in the present work a surface approximation of image height map using polynomial approximation is used.
Let accordingly I be the original image to be reconstructed (Fig. 1). Such an image has been obtained using a digital object whose virtual illumination has been provided under a CAD software environment. Using brightness as an elevation factor, it is possible to build a height map H for the original image (Fig. 2). Obviously, higher values in the height map are located in correspondence of the higher brightness values i.e., in the areas where scene illumination insists.
Once H is defined, by means of a least-square approximation using a cubic polynomial approximation it is possible to discard brightness contribute due to both high frequency details and illumination so as to retrieve a surface Γ resembling a very smoothed surface.
In Fig. 3 the surface Γ is over-imposed to the original (discrete) set of data.
The surface Γ strongly differs from the one obtainable using, for instance, a smoothing filter (see for instance Fig. 4 where a Gaussian filter is applied to the original image and a surface is then retrieved using brightness as surface height). Figure 5a shows the image Iˆresulting from the application of the proposed method; compared with the image obtained using a Gaussian filter (Fig. 5b) it can be visually noticed a more uniform gray level obviously due to the approximating surface Γ.

Retrieval of a Rough Solution R 1
Once the surface is obtained, it is possible to solve the SFS problem by minimizing the functional described by Equation 3 and using as an initialization function. The minimization procedure requires setting a series of Boundary Conditions (BCs). In the present work, a userbased procedure has been adopted for imposing such BCs. The description of the best BCs to be imposed for solving this kind of minimization problem (beyond the scope of the present work) has been extensively discussed in (Governi et al., 2013). On the basis of this work, the BCs taken into account are the following ones: Background, singular points (boundary white and white points) and silhouette contour (Fig. 6).
Such BCs are set by means of an interactive GUI (Fig. 7) developed using MATLAB® environment.
First, the user is required to select a single point in the image belonging to the object background. An automatic procedure has been implemented for detecting the overall background and setting the vertical unitnormal in all the background points so that they are built perfectly horizontal in the final 2.5D model.
Then, analogously to the BC taken for the background points, user has to set a specific unit normal vector for all the singular points (i.e., white points in image) and their boundaries. For such points the brightness level of the image reaches its maximum value (equal to 1); as a consequence the unit normal is set (by an appositely devised procedure) so that it coincides with vector L.
Finally, again by user interaction, it is necessary to set the value of the unit normal around its silhouette (i.e., on its outline), as inward or outward-pointing depending on the kind of surface, respectively convex or concave. This assumption, valid when the object represented may be clearly separated from the background (i.e., for all real shapes) is meant to determine the coarse volume of the shape. Once the BCs are set, the GUI allows inspecting (see the histogram positioned upper left in the GUI) the number of imposed conditions for each of the described BCs. Moreover, the devised interface allows to check the possible over-imposition of constraints; in case more different conditions are imposed to the same point, a procedure for removing the redundant BCs starts. Once the user is satisfied by the imposed BCs, the procedure provides, as output, a modified matrix formulation of the gradient where the number of unknown is reduced (since the unit normal for each point with imposed BC is obviously solved).
The modified formulation is the following Equation 9: Where E y (I), A y , B y and Φ are the "reduced" versions of, respectively, E (I), A, B and Φ. Minimizing Equation

Science Publications
JCS 9 means recovering 2.5D model for the smoothed image. For this aim, the regularizing factor is set equal to 1 while 2 λ is set equal to 10-2. As already mentioned, the main strength in minimizing the functional here is the use of the surface as initialization function. Minimization is carried out using a Gauss-Seidel iterative procedure with Successive Over Relaxation (SOR) method (Ikeuchi and Horn, 1981). Minimization procedure is stopped after a predefined number i of iterations whose value depends on the image size n x m. In particular the number of iterations is set equal to 0.1 x n x m. The final result of this step consists of a normal map Φ from which, using the approach described by (Tsai and Shah, 1994), it is possible to exactly retrieve the surface R 1 that generates a low-frequency version of the final, desired, surface. In Fig. 8 the surface obtained starting from image in Fig. 5a is provided.

Retrieval of a Rough Solution
Despite the rough solution R 1 visually resembles the overall geometry of the original image, the image filtering could have been, generally speaking, removed too much details. In order to partially overcome this possible drawback, another rough solution (R 2 ) is evaluated. Such a surface is retrieved by minimizing the functional of Equation 3 using a low value for (e.g., = 0.2) and a high value for (e.g., = 2). This approach allows to obtain a very smoothed surface (see Fig. 9).

Retrieval of the Rough Solution R as a Linear Combination of R 1 and R 2
Since two versions of a smoothed surface visually resembling the overall geometry of the original image are available, a final rough solution R can be obtained as a linear combination of R 1 and R 2 . In this work the weights of the linear combination are set equal to 0.5 so as the surface R coincides with the mean value between R 1 and R 2 .

Retrieval of the Final Surface
As already stated, one of the main strength of the proposed procedure rely in the fact that the minimization procedure can provide a more reliable solution if the iterative process is guided by initial guess of the final surface. In such terms, the lowfrequency surface R can be an effective initialization surface. Accordingly, once the surface R is obtained, it is possible to compute its normal map Φ R and to use it for initializing Equation 8. Minimization can then proceed using again a Gauss-Seidel iterative procedure with Successive Over Relaxation (SOR) method. The regularizing factors λ 1 and λ 2 are now balanced (e.g., both equal to 1). The resulting normal map is, eventually, converted in the final surface using the approach proposed by (Wu et al., 2008). In Fig. 10 the surface obtained for the exemplificative case of Fig. 1 is shown.

CASE STUDIES
The devised method has been tested against an extensive set of case studies in order to highlight strengths and possible weaknesses. In the present paper two of them are reported.

Case Study 1: Old Man's Head
The first case study consists of a grayscale image representing an old man's head detailed with several features such as: The wrinkles on the skin and the parietal branch of the superficial temporal artery on the temple (Fig. 11).
In Fig. 12 the height map of the original image is shown. Using the approach described above, the polynomial approximation Γ can be easily retrieved (Fig. 13).
Using Γ as initialization function it is possible to retrieve the surface R 1 (Fig. 14) using the approach described in section 2.3. By using the approach described in section 2.4. It is possible to retrieve surface R 2 (Fig. 15).
Finally, surface R is evaluated as a linear combination of R 1 and R 2 . Such a surface is shown in Fig.16. As it is clearly visible in Fig. 17a, the final results obtained by minimizing the functional of Equation 3 using as initialization function and balancing the regularizing factors closely resembles the expected surface. It has to be noticed that even finest details, as wrinkles or veins, have been correctly reconstructed. Conversely, the result obtained using image embossing-based techniques (Fig. 17b) is significantly worse thus demonstrating the effectiveness of authors' method with respect to such a traditional one.

Case Study 2: Darth Vader Helmet
The second case study is a synthetic image representing the legendary Darth Vader helmet (Fig.  19a). Using the proposed procedure, it is possible to retrieve the 2.5D model of Fig. 19b.
Since the ground truth is available for this case study, a color map of the absolute distance between the retrieved surface and such a ground truth (Fig. 14) has been produced in order to highlight the existing differences. The analysis of Fig. 19a and b points out that the proposed method provides good results in reconstructing 2.5D models from single shaded images. In particular, referring to the color map of Fig. 20 it can be observed that the final reconstruction generally resembles the original image. Moreover an absolute error within 10% is obtained for the second case study; this can be considered reasonable for this kind of representations.

CONCLUSION
The present work described an improved interactive method for single image based surface reconstruction obtained by improving the approach proposed in (Governi et al., 2014). The reconstruction was performed starting from the retrieval of a preliminary surface R roughly resembling the desired digital 2.5D model. Such a surface, created as a linear combination of two surfaces, one obtained by removing high frequency details from original image and the other one obtained using an interactive SFS-based Science Publications JCS method, was used as to initialize a properly defined functional in order to guide the minimization process.
Minimization has been carried out using an improved functional where both smoothness and brightness constraint are weighted using appropriate regularizing factors.
Results obtained for a set of case studies show that the proposed method provides effective results.
Once the user has set the boundary conditions using the devised GUI, the time the minimization algorithm takes to converge depends on the image size and on PC architecture. In order to provide an estimate of process time let's consider a PC with Intel Core i7 processor at 2.6 GHz and 16 Gb RAM. Results, in terms of convergence time, for differently sized images are shown in Table 1. As already mentioned, the number of iterations itself depends on image size and this is one of the main reasons while time increases exponentially. Moreover, computational time for each iteration increases due to the fact that the number of variables to take into account increases linearly with the number of unknowns. Nonetheless, for practical uses, since an image sized 500×500 can be considered a good starting point for 2.5D model reconstruction, about 6 minutes is a more than acceptable computational time.
Future works will be addressed in testing the procedure on more natural images and in implementing different minimization techniques also taking into account possible improvements in reducing computational time.

ACKNOWLEDGMENT
Authors wish to thank Prof. Alessandra Papini and Prof. Stefania Bellavia for giving us some hints about implementation strategies to be used in minimization techniques.