A New Separable Logarithmic Algorithm for Non-Linear Optimization 1

Problem statement: The idea of this study stemmed from the fact that most of the currently used optimization algorithms use a local quadratic representation of the objective function. It also arisen from the fact that the objective function may not be represented adequately by quadratic functions and the global minimizer may be obtained for objective functions. So, in this study, we generalized the field of quadratic model into the field of the non-quadratic model. Approach: A new non-quadratic model was suggested for solving unconstrained optimization problems, which modified the classical Conjugate Gradient (CG) algorithm by scaling the standard quadratic model. Results: The new algorithm was derived and evaluated theoretically and numerically for some standard well-known and effective test functions. The results, in general, indicated that the new algorithm had improvements on different well-known algorithms used in this study. Conclusion: The new proposed algorithm would be generic and easy to implement in all gradient based optimization process. Its simulation results showed that it was robust and had a potential significantly enhance the computational efficiency of iterations and function evaluations.


INTRODUCTION
This study considers the calculation of a local minimizer x* say, for the problem Minf(x), x∈R n , f∈C 2 (1) It is assumed that expressions for the elements of the gradient vector g(x) = gradient of f(x) are available are calculated but the Hessian matrix is not available. Standard algorithms for solving this problem include CG-algorithms which are iterative algorithms and generate a sequence of approximations to minimize a function f(x). This type of algorithms requires 4n locations of computer storage to implement. However, this study considers a more general model than the usual quadratic function for i j ≠ . Hestenes and Stiefel (1952) published the first CGalgorithm, for solving a system of linear algebraic equations. Fletcher and Reeves, (1964) were the first, among other scholars, to use this technique to minimize a non-linear function of several variables. Since then this algorithm has been used successfully to tackle many problems.
We can list outline of the CG-algorithm as follows: Outline of the standard CG-algorithm: for given TOL and x 0 ∈R n an initial estimate of the minimizer x*: Step 1: Set 0 0 d g = − Step 2: For i = 1, 2,…, compute i Step 4: Calculate the new direction d i = −g i + β i d i−1 where β I is the conjugacy coefficient.
It is considered as one of the following formulas: i i i 1 i 1 i 1 Called: Al-Bayati and Al-Assady (BA). For the details of these formulas (Fletcher, 1987;Al-Bayati and Al-Assady, 1994).
When, quadratic functions and exact line searches are used, all the above formulas for i ′ β s are equivalent.
However, these formulas vary according to general functions.
CG-algorithms have, in general, the following: BASIC properties: • The conjugacy condition • The orthogonally condition • The descent direction • The quadratic termination condition with Exact Line Search (ELS)

Extended CG-algorithms for non-quadratic models:
In this study, a more general model than quadratic one is suggested as a basis for a CG-algorithm. If q(x) is a quadratic function, then a function F(q(x)) is defined as a non-linear scaling of q(x) if the following condition holds: where, x* is the minimizer of q(x) with respect to x, (Spedicato, 1976). The following SCALING properties for f(x) are immediately derived from the above condition: • Every contour line of q(x) is a contour line of f(x) • If x* is a minimizer of q(x), then it is a minimizer of f(x) • If x* is a local minimizer of q(x), then it is a local minimizer of f(x) Boland et al. (1979) was the first who observed that q(x) and f(q(x)) have determined the same search directions so that the finite termination property for their algorithm was satisfied. A CG-algorithm which minimizes the function f(x) = q(q(x)) p , p > 0 and x∈R n (5a) in at most n step was described by (Fried, 1971). The special case: where, ε 1 and ε 2 are scalars, was investigated by (Boland et al., 1979). (Tassopoulos and Storey, 1984) proposed two different rational models defined by: where, ε 1 and ε 2 are scalars and: Al-Bayati (1993) proposed another rational model defined by: In this study, a new logarithmic model is investigated and tested on a set of several standard test functions, on the assumption that condition (1) holds. An extended CG-algorithm is developed which is based on this new model and scales q(x) by the natural log function for the rational q(x) functions: We first observe that q(x) and F(q(x)) given by (6) have identical contours, though with different function values and they have the same unique minimum point denoted by x*.
New ECG-algorithm for non-quadratic model: For any f satisfying the condition (4), it is shown in (Boland et al., 1979) that the updating process given below generates identical conjugate directions and the same sequence of approximations x i to the minimizer x*, as does the original algorithm of (Fletcher and Reeves, 1964) when applied to f (x) = q(x). Now let us modify the last scaling property of the function f(x) by: If x* is a global minimizer of q(x) implies that it is a global minimizer of f(x), in the new extended logarithmic model defined in (6). Recall the basic theorem of (Renpu, 1989) which was illustrated below.
Suppose that F(x) has the form: and: f 1 (x 1 ) > 0 and g 2 (x 2 ) > 0 (8) It follows from (7) that: Equation 9 is a separable function according to the following theorem. Proof: We can conclude that x* is a global minimizer of log F(x) if and only if x 1 * and x 2 * are respectively global minimizer of log f 1 (x 1 ) and -log g 2 (x 2 ). Furthermore, the monotonicity of the log function implies that x* is a global minimizer of F(x) if and only if x 1 * and x 2 * are respectively global minimizer of f 1 (x 1 ) and g 2 (x 2 ) (Renpu, 1989). Before listing the main outline of the new proposed algorithm, let us consider first the main outline of the (Tassopoulos and Storey, 1984) and (Al-Bayati, 1993) rational model extended CG-algorithms.
Step 3: Set i (a / w) ρ = 2 Step 4: Calculate the new direction Outline of the new ECG-algorithm: Step 1: Step 2: For i = 1, 2 ... compute x i = x i-1 + λ i−1 d i−1 where λ i−1 is the optimal step-size obtained by Armijo linear search procedure: set scalars s k , β and σ with: i.e., m= 0,1 are tried successively until the inequality above is satisfied for m = m k .
Step 3: Calculate f which is a function value of an actual iterate as: exp(f) =1 + f + f 2 /2! + … and define: Step 4: If |w|<0.1 E-5 or |c|<0.1 E-5, then set ρ i = 1.0 and go to Step 6. Else continue Step 5: Compute: where, the derivation of scaling ρ i will be presented later.
Step 6: Calculate the new direction d where, β i is defined by different modified formulas as follows: where, β i in (10) are modified FR, HS, PR, DD and BA respectively. CG-algorithms are usually implemented by restarts in order to avoid an accumulation of errors affecting the search directions. It is therefore generally agreed that restarting is very helpful in practice, so we have used the following restarting criterion in our practical investigations. If the new direction satisfies: then a restart is also initiated and the new direction is sufficiently downhill.

The derivation of the new ECG-algorithm:
The implementation of the extended CG-algorithm has been performed for general functions F(q(x)) of the form of (6). The unknown quantities ρ i were expressed in terms of available quantities of the algorithm (i.e., function and gradient values of the objective function).
It is first assumed that neither ε 1 nor ε 2 is zero in Eq. 6. Solving (6) for q(x), yields: and using the expression for ρ i (henceforth NEW the quantity that has to be determined explicitly is (ε 1 /ε 2 ). During every interaction (ε 1 /ε 2 ) must be evaluated as a function of known available quantities from the relation: where, G is the Hessian Matrix and x* is the minimum point. Now, we have: Since g T i d i−1 = 0, therefore, we can express NEW i ρ as follows: From (14) and (15), we get: The quantities (q i−1 /q i ) and f ' i q i can be rewritten as: Substituting (18) and (19) in (17), gives: Using the transformation: and from RHS of (13) ; (20) and (21), we have:

Proof:
For the new algorithm set It follows from (26) and (28): By (25) and the definition of s k , we have: Set min{ , (1 ) / L} η = σ βσ − σ , by (29) and (30), we have: Proof: Suppose that the theorem is not true, then ∃ a constant α>0 such that: Now, for our new search direction defined in Step 6 of the new extended algorithm we have: Now, for any k≥1: which is a contradiction. Hence, the theorem must be true and the algorithm has super-linear convergence
We conclude that our new proposed rational logarithmic model is superior to some models in both quadratic and non-quadratic models.

DISCUSSION
In this study, we have introduced a new nonquadratic CG-algorithm which is based on log of two rational functions. The new algorithm is compared with three well-known algorithms; namely FRCG; TS, (Tassopoulos and Storey, 1984) and B; (Al-Bayati, 1993) algorithms using seven well-known non-linear test functions with different dimensions taken from (Al-Bayati, 2007). Our numerical results indicate that the new technique has an improvements of about (-15-98%) in NOI against the others, while it saves about (6-93%) in NOF against the others.

CONCLUSION
In this research, a new fast extended CG algorithm is introduced. The proposed algorithm improved the standard FRCG; TS and B algorithms by adaptively modifying the search direction. The new proposed algorithm is generic and easy to implement in all gradient based optimization process. The simulation results showed that it is robust and has a potential significantly enhance the computational efficiency of iterations and function evaluations.