Variable Neighborhood Simplex Search Methods for Global Optimization Models

Problem statement: Many optimization problems of practical interest a re encountered in various fields of chemical, engineering and managem ent sciences. They are computationally intractable. Therefore, a practical algorithm for s lving such problems is to employ approximation algorithms that can find nearly optimums within a r easonable amount of computational time. Approach: In this study the hybrid methods combining the Var iable Neighborhood Search (VNS) and simplex’s family methods are proposed to deal with the global optimization problems of noisy continuou s functions including constrained models. Basically, the simplex methods offer a search scheme without the gradient information whereas the VNS has the be tter searching ability with a systematic change of neighborhood of the current solution within a local search. Results: The VNS modified simplex method has a better searching ability for optimization pro blems with noise. The VNS modified simplex method also outperforms in average on the characteristics of intensity and diversity during the evolution of design point moving stage for the constrained optimization . Conclusion: The adaptive hybrid versions have proved to obtain significantly better results than the conventional methods. The amount of computation effort required for successful optimization is very sensitive to the rate of noise decrease of the pro cess yields. Under circumstances of constrained optimiza tion and gradually increasing the noise during an optimization the most preferred approach is the VNS modified simplex method.


INTRODUCTION
Global optimization problems consist of functions on an open and compact set containing some problem variables. In various cases of practical interest the global optimization is very difficult because there is the presence of many local optima. The number of which tends to exponentially increase with the problem dimension. There are two general algorithms for finding the global optimum. Firstly, a set of random starting design points is generated for a use of conventional algorithms to approach the optimum via enumerative search strategies. A large number of starting design points have to be tried in order to achieve a sufficiently large probability of finding the global optimum. These sequential procedures are time consuming and become intractable as there is an increase of the dimensionality of the problem. Secondly, the algorithms perform the new design points to avoid entrapments in local optimum and continue search to approach near-optimal solutions. The methods of this type include simplex's family and Variable Neighborhood Search (VNS) methods.
The simplex's family methods are robust to lower levels of inaccuracies or stochastic perturbations in function values. It only uses the ranks of the function values to determine the next move, not from the function values. However, the global optimization searching ability depends on whether the initial design point is located on the nearby of the design point or not. They own the merit of a better search speed for a local optimization problem and results in the deficiency of globally searching ability. The VNS is a technique that exploits the idea of neighborhood structure change in a systematic way, both in the descent to local optima and in the escape from the valleys which contain them tries to escape from a local optimum by changing the neighborhood structure. After obtaining a neighboring solution to the current solution, the VNS executes a local search until a local optimum is reached and moves to it if there has been an improvement in modifying the neighborhood structure. Otherwise, there is a systematic change in the neighborhood.
Effective optimization algorithms depend on their accuracy and their searching ability for global optimum.
The VNS has been applied to the computer science field in order to enhance the global optimization searching ability (Aungkulanon and Luangpaiboon, 2010). While the simplex's family methods and the VNS are both categorized into the primitive stage, that is, both of them are a direct search scheme without gradient information. Thus, it has a fast searching ability and has been widely applied to improve the design point of the global optimization problems (Gao et al., 2008). However, there are some disadvantages among both methods. The better searching ability for the global optimization problem and the characteristics of intensity and diversity during the sequential procedures are then expected after the integration.
The objective of this study is to investigate the choice of the best simplex design based method with the fine-tuning characteristic of the VNS for optimization of nonlinear continuous unconstrained functions and constrained models. This study describes various selected simplex's family methods of the modified simplex, super modified simplex and weighted centroid simplex including the variable neighborhood search method. Experimental results and analyses are performed to compare the performance of the proposed variable neighborhood simplex search methods. The conclusion is also summarized and it is followed by acknowledgment and references.

Modified Simplex Method (MSM):
The simplex algorithm is based on the geometry and the simplex design is composed of a k-dimensional polyhedron with k+l equidistant design points or linearly independent vertices. For k equal to two, this simplex is an equilateral triangle composed of 3 vertices in 2dimension space, for k equal to three any four design points in a different plane can form a tetrahedron. Therefore, according to the k-dimensional problem, a simplex is a geometrical figure consisting, in k dimensions, of (k+1) design points and then, the objective function of those design points can be obtained and compared. After each step, decisions are required to determine in which direction the simplex must move.
The new symmetrical simplex consists of one new point and k design points from the previous simplex or discarding the worst point and replacing it with a new point. Repetition of simplex reflection and objective function measurement form the basis for the most elementary simplex algorithm. This iterative process is based on gradually moving away from the experiment with the worst result in a simplex toward the optimum. The rigid simplex method appears to have been first proposed by Spendley et al. Many modifications to the original simplex method have been developed. Nelder and Mead (1965) modified a rigid simplex method to allow various procedures to adapt to the global optimization much more readily. This method is referred to modified simplex method (MSM). MSM allows the simplex to converge more rapidly towards an optimum by expansion and multiple ways of simplex contraction along the line of conventional reflection in order to speed up the convergence (Fig. 1). When the result becomes more preferable, an expansion with a preset coefficient is performed, to stretch the move beyond the simple reflection.
In some cases when the outcome is more desirable than the worst one, but still worse than all the remaining outcomes, a contraction with a preset coefficient level is used to shorten the move in comparison to the reflection. Moreover, massive contractions are performed when the new result gets worse than any of the previous ones. In this case the size of the simplex is reduced by contracting each of its edges to one half of its previous length toward the vertex producing the best outcome. A new simplex is thus generated with k new measurements and the sequential optimization procedures are repeated. When the contraction approaches to the convergence criterion, this replacing process stops, that is, the iteration stops (Fig. 2).

Weighted Centroid Simplex Method (WCSM):
Ryan et al. introduced the weighted centroid simplex method. In order to reflect the different influence or different design points of the simplex, each point is assigned a weight, which is equal to the ratio of the difference of objective function values to their geometrical distance between this point and the point to be reflected. In doing so, each point is treated differently according to its contribution to the potential reduction of the objective function. In general, for a k-variable problem, the direction for the next search is obtained by combining those weights in the simplex. WCSM proceeds in a manner entirely analogous to MSM ( Fig. 2) with the only exception of WC. The centroid C is replaced by the weighted centroid WC of hyperface opposite W in the reflected equation by: The W-R vector of is terminated at the design point where the boundary constraint is intersected by the vector and the design point is then considered an effective R (R'). The yield is evaluated at R', the second polynomial curve is fitted through the yields at W and R' and the new vertex location determined.
However, the predicted optimum should occur at or very near the Centroid (C) of the new vertex. At this point it would reduce the dimensionality of the process and would virtually terminate any further progress of the simplex design in one or more dimensions. Thus, a safety interval controlling such a move is used to locate the new vertex with a percentage of the factor domain away from the centroid.

Variable Neighborhood Search (VNS):
In order to solve the problem presented in this study, the simplex's family methods are modified and designed by the metaheuristic strategies of Variable Neighborhood Search (VNS) initially introduced by Mladenovic and Hansen (1997). The solutions obtained with these hybrid methods are compared with the ones obtained with what may be the conventional methods for obtaining the optimum (Imran et al., 2009). The strategy for the VNS involves iterative exploration and tries to escape from a local optimum by changing the neighborhood structure in a systematic way to approach the optimum.
Each time in a local search routine for optimization the basic idea of VNS meta-heuristic is to use more than one neighborhood structure and to proceed to a systematic change of them within a local search. VNS remains in the same design point until another solution better than the incumbent is found and then jumps there. The method, without forbidden moves, then escapes from the current solution to a new one. Neighborhoods are usually ranked in such a way that intensification of the survey around the current design point is naturally followed by diversification controlled by a set of parameters. VNS also allows a controlled increase during the iterative improvement. The pseudo code of the VNS is shown in Fig. 4.   (Zhao et al., 2009). These examples have been previously solved using a variety of other techniques, which is useful to demonstrate the validity, effectiveness and robustness of the proposed variants on the variable neighborhood simplex search. There are 100 realizations in each model. For each model, the Key Performance Indices (KPI) of central tendency, dispersion and robustness are represented by the Average (Av), Standard deviations (Sd) of these objective function values and the Taguchi Signal to Noise ratio (SN). Signal to Noise ratio will present sensitivity of response to noises or uncontrollable factors. This ratio is used to point out stability of the design system and quality of chosen design's factors. The minimum (Min) and maximum (Max) of the yields are also included for the constrained models.
On the first scenario, eight stochastic non-linear unconstrained mathematical functions in the context of response surface with two variables were used to test performance measures of the related methods whilst searching for the maximum. They consist of Parabolic (PA), Shekel (SH), Rosenbrock (RO), Branin (BR), Camelback (CA), Goldstein-Price (GO), Rastrigin (RA) and Styblinski (ST) functions. Each of these test functions has a unique nature. The test functions were randomized by adding a normal and independent distributed error term or noise with zero mean and standard deviation of 0.0, 1.0, 2.0 and 3.0. Each optimization run uses independent random number streams. Secondly, four constrained mathematical models are applied to determine the best parameter choice as follows.
Constrained model # 1: It is assumed that an initial design point of variables which satisfies all the constraints is available. A design point selected must satisfy the explicit constraints, but need not satisfy all the implicit constraints. If an implicit constraint is violated or a design point is found to be infeasible, a new design point is created using the previously generated feasible design points.

RESULTS
The hybrid algorithms of simplex's family methods and the VNS are compared with respect to accuracy, consistency and computational effort. For all stochastic functions and for the manufacturing models, there is an overall significant difference between the hybrid optimization methods with respect to accuracy for both the smallest errors and higher levels of the errors of the optimization runs. For the manufacturing models the Variable Neighbourhood Simplex Search (VNSS) based methods of VMSM (I), VWCSM (II) and VSMSM (III) were tested using both one large simulation run and have smaller simulation runs to evaluate a design point. There is no statistically significant difference in accuracy or consistency among them. The results for the stochastic functions with respect to the accuracy, consistency and computational effort of the hybrid methods when considering the errors at the lowest and highest levels of 0.0 and 3.0 are shown in Table 1 and 2, respectively. The overall results on all hybrid methods are summarized in Table 3.     I  II  III  I  II  III  I  II  III  PA  SH  RO  BR  CA  GO  RA  ST When there is no noise the selected performance measures of hybrid methods are not statistically different when compared. Generally, the average (Av), the sample Standard deviation (Sd) and the Signal to Noise ratio (SN) of actual objective function values deteriorate when there is an increase of noise on the objective function values as shown in Fig. 5-7, respectively (Lan, 2010;2011;Kumar et al., 2009). However, on all performance measures the VMSM performs better than other versions of VNSS based methods. When the Sd and SN performance measures are considered the preferred methods seem to be the VSMSM and VWCSM, respectively.
Moreover, the controllable parameters of the simplex's family methods such as the simplex size, an increase of simulation size or replicates, can even be chosen in such a way that the hybrid methods perform substantially better than the conventional method with respect to accuracy and consistency.  I  II  III  I  II I  II  III  I  II I  II  III  I  II I  II  III  I  II  III  Av -208.990 -200.550 -200.740 -204.150 -198.760 -200.070 293.860 568.040 293.920 The improvement found for any hybrid method depends on the nature of the objective function. On the rather complex function such as the combined multi-peak and curved ridge surface the hybrid methods perform better. For all hybrid methods, the improvement in accuracy and consistency is achieved at the cost of an increase in the computational effort. For most objective functions we found that the VMSM needs less additional computational effort than the other hybrid methods. We conclude that the VMSM is effective modification of the simplex's family methods in view of its accuracy and computational effort. For a study of the constrained models, the numerical results are given in Table 4-7. Generally compared to the benchmark optimization and approximation algorithms, the VNSS based methods need more computational effort than the algorithms that use the same criterion but an increase of the number of simulation runs were mainly used to evaluate a vertex for the success of all the related constraints. We found that applying the VMSM can lead to substantially more accurate and consistent results without the cost of extra computational effort when compared to other hybrid methods. However, if optimizing the manufacturing models using the conventional algorithms would lead to at least the same results as optimizing with the VNSS based methods then there would be no point in using the hybrid algorithms.

DISCUSSION
The proposed hybrid methods are the Variable Neighborhood Simplex Search (VNSS) based methods. These VNSS based methods adopt the design point evolution process of the VNS via the systematic neighbourhood change and introduce the searching mechanism of the simplex's family methods for generating the new generation. With the VNS evolution, "in the descent to local minima and in the escape from the valleys", the proposed hybrid methods combine the evolution method and the simplex design based methods with respect to their own merits to deal with the global optimization.
The VMSM is useful for the optimization of stochastic functions. However, the amount of noise in the functions largely determines the success of the optimization procedure. The study described in this research is designed to improve the performance of the VNSS based methods when applied to the optimization of very noisy functions. Several conventional adaptive extensions after Nelder and Mead which are the super modified simplex and weighted centroid methods are tested using stochastic functions and representative constrained models of simulated manufacturing processes. These conventional adaptive extensions include algorithms that combine the weights in the simplex and fit a second order polynomial curve through all relevant vertices during an optimization run.
We also compared the extended hybrid methods to benchmark algorithm based on the original methods. We found that a relatively simple extension of the VMSM simplex method is able to detect when noise obstructs the optimization process and consequently, indicate the moments when the evolution process of the VNS is needed to successfully continue the constrained models. For the constrained models and for most of the stochastic functions, gradually increasing the realization or the number of simulation runs used for a function or model evaluation, can lead to considerable improvements in accuracy and consistency of the observed optimal design points.
The amount of computational effort required by all hybrid methods, which apply to constrained models, is less dependent on the number of constraints. The evaluation of manufacturing models is often very timeconsuming and the VMSM is to be preferred over the use of the original benchmark algorithm with the context of the Box complex method and an expensively large simulation size. In the tests described in this research a maximal number of evaluations to end the optimization process are applied.
However, in applications, the use of more sophisticated termination criteria should be concerned. Furthermore, the choice of the initial simplex size should be also considered. It could emphasize the fact that the simplex's family methods are the local search method, no guarantee is given for finding the global optimum. Therefore, multiple starting design points and/or multiple searches from the same starting design point should always be tried. However, in this study there is no experiment on the effect of previous concerns, since the performance of single applications of the hybrid methods was primarily of interest.

CONCLUSION
In this study, we present new approaches based on the variable neighborhood simplex search for solving both stochastic and simulated process models. The methodology combines and extends the attractive features of both simplex's family methods and the VNS. There are two interesting issues to be investigated. Firstly, it is the determination of the robust-to-noise methods that apply their adaptive measures to all vertices used during iterative processes. Secondly, robust-to-constraint approach can also be chosen among modified methods. Heuristics main aims are to provide, in reasonable time, near-optimal solutions to stochastic, constrained or unconstrained optimization problems and to accelerate many steps via knowledge discovery and systematic systems.
Experimental results show that the VMSM is practically efficient to find the optimum on both interesting issues and may be very helpful in studying optimization problems in actual engineering systems. The VMSM adopts the search approach of the simplex method and introduces the variable neighborhood search concept and thus, this can have the greater populations for searching. Besides, the VMSM has the expansion and contraction operations with the reflection of the different influence or different design points to search for the global optimization so that the VMSM can provide designer various possible strategies.
In addition to considering alternative improvements of the variable neighborhood simplex search based methods, the question how this method compares to other hybrid algorithms with recent stochastic element of hunting search and harmony search in the optimization of stochastic objective functions and constrained models remains to be addressed as well. In addition to considering hybrid improvements of the simplex's family methods with other recent metaheuristics, the question how these simplex's family methods depend on the parameters such as the process of the additional re-evaluation measures of problem objective function values, the multiple initial simplex design points and the process of increasing simulation size or number of replicates, could be also investigated to deal with the dynamics information during optimization runs.