New Scaled Sufficient Descent Conjugate Gradient Algorithm for Solving Unconstraint Optimization Problems

: Problem statement: The scaled hybrid Conjugate Gradient (CG) algorithm which usually used for solving non-linear functions was presented and was compared with two standard well-Known NAG routines, yielding a new fast comparable algorithm. Approach: We proposed, a new hybrid technique based on the combination of two well-known scaled (CG) formulas for the quadratic model in unconstrained optimization using exact line searches. A global convergence result for the new technique was proved, when the Wolfe line search conditions were used. Results: Computational results, for a set consisting of 1915 combinations of (unconstrained optimization test problems/dimensions) were implemented in this research making a comparison between the new proposed algorithm and the other two similar algorithms in this field. Conclusion: Our numerical results showed that this new scaled hybrid CG-algorithm substantially outperforms Andrei-sufficient descent condition (CGSD) algorithm and the well-known Andrei standard sufficient descent condition from (ACGA) algorithm.


INTRODUCTION
For solving the unconstrained optimization problem: where, 2 f : R R → is continuously differentiable function, bounded from below.Starting from an initial guess, a nonlinear CG-algorithm generates a sequence of points {x k }, according to the following recurrence formula: where, α k is the step-length, usually obtained by Wolfe line searches:

Algorithms based on sufficient descent conditions:
This type of algorithms present a modification of the standard computational CG scheme in order to satisfy both the sufficient descent and the conjugacy conditions in the frame of CG as in (4), with: g g y g ( )( ) ( ) y g s g g g y s y s Equation 4-7 are represent an algorithm that belongs to the family of scaled CG-algorithms introduced by (Birgin and Martinez, 2001).Observing that if f is a quadratic function and α k is selected to achieve the exact minimum of f in the direction d k then T k k 1 s g 0 + = and the formula (5) for CGSD k β reduced to the Dai and Yuan computational scheme (Andrei, 2008a): However, the parameter CGSD k β is considered for general non-linear functions and inexact line searches and it is selected in such a manner that the sufficient descent condition is satisfies at every iteration.Besides, the parameters CGSD

CGSD algorithm (Andrei, 2007):
Step 1: Initialization.Select Step 4: Compute ACGA algorithm (Andrei, 2009a): This algorithm also presents a modification of the Dai and Yuan computational scheme in order to satisfy both the sufficient descent and the conjugacy conditions in the frame of CG.The steps of this algorithm are same as in Andrei (CGSD), except Step (4) which will be defined as: y g (y g )(s g ) y s (y s ) Or: Now we introduce a new proposed method based on modifying both scalars θ k+1 and β k .
A new proposed algorithm (say, new hybrid): here, we are going to investigate another new sufficient descent algorithm based on the reformulation of the scalars θ k+1 and β k .The new proposed scalars are depend on the general hybrid techniques of two or more than two parameters.These scalars are very useful in making the search directions generated by the new algorithm more sufficiently descent.The outlines of the new proposed algorithm are given by: Step 1: Select y g s g g g y s y s y g s g y g y s y s where the details of ACGA k β are given in (Andrei, 2009a).
Step 5: If: The algorithms ( 12) and ( 13) belongs to the family of hybrid CG-algorithms.

Rate of convergence of the new hybrid algorithm: Theorem
is given by (5), then: Proof: , which satisfy (15).Multiplying (14e) by T k 1 g + , we have: g y s g s g y s Using (17) in ( 16): y s y s We get: Hence, the direction given by (3) and ( 7) is a descent direction.If for all k, CGSD for CGSD method observe that: Where: ) s y s y s From ( 20) and ( 22) we get: Now, by summarization of CGSD k 1 Q + as: And considering the conjugacy condition: ( ) ( ) After doing some algebraic operations, we get: ) ) 0 y g y s y g (y s ) )( ) 0 y g y s g + in the numerator of CG operators has a strong global convergence (Al-Bayati et al., 2009), hence from the first bracket of the Eq.29b: From (29c) we have observed that: then for all k the search direction d k+1 given by ( 3) and ( 7) with (33), given later, satisfy the sufficient descent condition.
Anticipative θ θ θ θ k+1 derivation: Recently (Andrei, 2004) using the information in two successive points of the iterative process, proposed another approximation scalar to the Hessian matrix of function f , to obtain a new algorithm which was favorably compared with the Barzilai and Browein's method.This is only a half step of the spectral procedure.Indeed, at the point , we can write: where, z is on the line segment connecting x k and x k+1 .
Having in view the local character of the searching procedure and that the distance between x k and x k+1 is small enough, we can choose z = x k+1 and consider . This is an anticipative viewpoint, in which a scalar approximation of the Hessian at point x k+1 is computed using only the local information from two successive points: x k and x k+1 , therefore we can write: This formula can also be found in Dai an Yuan (Andrei, 2008b).Observing that k 1 0 + γ > for convex functions (Andrei, 2007) In this cases, the idea is to reduce the step size α k as k k , α − η maintaining the other quantities at their values in such away so that γ k+1 is positive.To get a value for η k , let as select a real µ>0, "small enough" but comparable with the value of the function and have: for which a new value of γ k+1 can be computed as: g d ]  d d ( ) with these, the value for parameter θ k+1 is selected as: where, γ k+1 is given by either (31c) or (31e).
Proposition: Assume that f(x) is continuously differentiable and f (x)

∇
is Lipschitz continuous, with a positive constant L. then at point x k+1 : Proof: From (31c) we have: where, k ζ is on the line segment connecting x k and x k+1 .Therefore: Using the inequality of Cauchy and the Lipschitz continuity it follows that: Therefore, from (31f) we get a lower bound for is given by Eq. 12c, then: [ 2 (g s )g ] (y s ) Using Eq. 37 in 34 we get Eq.33.
Hence, the direction given by ( 12 where: where the matrix ACGA Now, by summarization of ACGA k 1 Q + to resemble the Quasi-Newton method, as: and considering the conjugacy condition: ( ) ( ) g s y [I s s ]g 0 y s y s After doing some algebraic operations, it follows that: y s g y (g y )(y s ) g s g g (g s ) Therefore using (45) in (12c) we get (12b).

RESULTS
We present the computational performance of a Fortran implementation of the new hybrid algorithm on a set of 1915 unconstrained optimization test problems/dimensions.The Fortran implementation of the present algorithm is based on the Fortran 90 implementation of the scaled CG-method provided by (Birgin and Martinez, 2001).The comparisons of algorithms are given in the following context.Let ALG !i f and ALG 2 i f be the optimal values found by ALG1 and ALG2, for problem i = 1.., 65, respectively.We say that, in a particular problem i, the performance of ALG1 was better than the performance of ALG2 if: and the number of iterations, or the number of functiongradient evaluations, of ALG1was less than the number of iterations, or the number of function-gradient evaluations of ALG2, respectively (Andrei, 2009a;2009b;2008c).We compare the performance of our new hybrid algorithm against the CGSD-algorithm (Andrei, 2008a) and against the standard ACGAalgorithm (Andrei, 2009a) in three different tables.In  1, subject to #iter, the new hybrid was better in 163 problems while CGSD was better in 134 problems; they are equal in 293 problems and fail in 60 problems out of 650 problems; now subject to #fgev, the New Hybrid was better in 168 cases while CGSD was better in 138 cases; they have equal results in 284 cases and fail in 60 cases.
In Table 2, according to #iter, the new hybrid algorithm was better in 274 cases while ACGA was better in 194 cases; they have equal results in 155 cases and fail in 54 cases.However, according to #fgev, New Hybrid was better in 273 cases while ACGA was better in 209 cases; they have equal results in 114 cases and fail in 54 cases.
set k = k+1 and continue.Step 3: Compute the line search parameter α k which satisfy the Wolfe-conditions: = k+1 and continue.Step 3: Compute the line search parameter α k which satisfy the Wolfe-conditions defined by (9) and(10).Update k 1 k k k x = x + d
and the line searches satisfy Wolfe conditions, then the search directions given by (3) and (7) satisfy the sufficient descent condition since Andrei's algorithm bound by: e., it is bounded away from zero.

Table 1
Table1and 2 present the performances of these algorithms subject to the minimum number of iterations (# iter) and the minimum number of functiongradient evaluations (# fgev).When comparing the new hybrid against CGSD in Table

Table 1 :
Performance of the new hybrid versus CGSD;

Table 3 :
Comparison of Different CG-algorithms with an arbitrary selection of 6 different test functions out of 65-test problems CGSD New hybrid ACGA -