A New Newton’s Method with Diagonal Jacobian Approximation for Systems of Nonlinear Equations

Problem statement: The major weaknesses of Newton method for nonlinea r equations entail computation of Jacobian matrix and solving s ystems of n linear equations in each of the iterations. Approach: In some extent function derivatives are quit costly and Jacobian is computationally expensive which requires evaluation (storage) of n×n matrix in every iteration. Results: This storage requirement became unrealistic when n becomes large. We proposed a new method that approximates Jacobian into diagonal mat rix which aims at reducing the storage requirement, computational cost and CPU time, as we ll as avoiding solving n linear equations in each iterations. Conclusion/Recommendations: The proposed method is significantly cheaper than Newton’s method and very much faster than fixed New ton’s method also suitable for small, medium and large scale nonlinear systems with dense or spa rse J cobian. Numerical experiments were carried out which shows that, the proposed method is very e ncouraging.


INTRODUCTION
Consider the system of nonlinear equations: where, F(x) : R n → R n with the following properties: • There exist x* with F(x*) = 0 • F is continuously differentiable in a neighbourhood of x* • F F'(x*) = J (x*) 0 ≠ The most well-known method for solving (1), is the classical Newton's method. However, the Newton's method for nonlinear equations has the following general form: Given an initial point x 0 , we compute a sequence of corrections {s k } and iterates {x k } as follows: Algorithm CN (Newton's method): where, k = 0, 1, 2... and J F (x k ) is the Jacobian matrix of F, then: Stage 1: Solve F k J (x ) s k = -F(x k ) Stage 2: Update x k+1 = x k + s k Stage 3: Repeat 1-2 until converges.
The convergence of Algorithm CN is attractive. However, the method depends on a good starting point (Dennis, 1983). Newton's method will converges to x* provided the initial guess x 0 is sufficiently close to the x* and J F (x*) ≠ 0 with J F (x) Lipchitz continuous and the rate is quadratic (Dennis, 1983), i.e.: For some h. Even though it has good qualities, CN method has some major shortfalls as the dimension of the systems increases which includes (Dennis, 1983) for details): • Computation and storage of Jacobian in each iteration • Solving system of n linear equations in each iteration • More CPU time consumption as the equations dimension increases There are several strategies to overcome the above drawbacks. The first is fixed Newton method, i.e., by setting J F (x k ) ≡ J F (x 0 ) for k>0. Fixed Newton is the easiest and simplest strategy to overcome the shortfalls for systems of nonlinear equations and it follows the following steps: Algorithm FN (fixed Newton): Let x 0 be given: Step 1: Solve J F (x 0 )s k = -F(x k ) Step 2: Set x k+1 = x k + s k for k = 0, 1, 2... FN method diminishes both the computation of the Jacobian (except for the first iteration) as well as avoiding solving n linear system in each iteration but is significantly slower (Natasa and Zorna, 2001). The second strategy is inexact Newton method. This method avoids solving Newton equation (Stage 1 of Algorithm CN) by taking the correction {s k } satisfying (Dembo et al., 1982;Eisenstat and Walker, 1985): Inexact Newton method is given by the following algorithm: Algorithm INM (Inexact Newton): Let x 0 be given: Step 1: Find some s k which satisfies: Step 2: Set: x k+1 = x k + s k where, {η k } is a forcing sequence. Letting η k ≡ 0 it gives Newton method.
Another modification is quasi-Newton's method, the method is the famous method that replaces derivatives computation with direct function computation and also replaces Jacobian or its inverse with an approximation which can be updated at each iterations (Lam, 1978;Denis, 1971). There are quite many modifications introduced to conquer some of the shortfalls (Drangoslav and Natasa, 1996;Hao and Qin, 2008;Natasa and Zorna, 2001), but most of the modifications requires to computes and store an n×n matrix (Jacobian) in each iterations (Natasa and Zorna, 2001). In some cases when the number of equations is sufficiently large it becomes computationally expensive and requires evaluation (and storage) of generally fully populated J F (x k ) of dimension n×n which requires more CPU time, that is why Newton method cannot handle large-scale system of nonlinear equations. In this study we propose a method that reduces computational cost, storage requirement, CPU time and also eliminates the need for solving n linear system in each iteration. This is made possible by approximating the Jacobian into diagonal matrix. The proposed method is significantly cheaper than Newton method, so much faster than Fixed Newton's method and is suitable for both small, medium and large scale systems of equations.

A new Newton method with diagonal Jacobian:
Consider the Taylor expansion of F(x) about x k : Then the incomplete Taylor series expansion of F(x) is given by: In order to incorporate correct information on the Jacobian matrix to the updating matrix, from (4) we impose the following condition (Albert and Snyman, 2007): is an approximated F evaluates at k 1 x + . Then (4) turns into: Hence we have: We propose the approximation of where k D is a given diagonal matrix, updated at each iteration. Then (7) turns to: Since we require D to be a diagonal matrix, says D = diag(d 1 , d 2 ,..., d n ), we consider to let components of the vector as the diagonal elements of D k , from (9) it follows that: Hence: for i = 1, 2, . . ., n and k = 0, 1, 2, . . ., n. Where: We use (14) (to safeguard very small (i) We propose the update for our proposed method (NDJ) as below: where, k D is defined by (11), provided Algorithm NDJ: Consider F(x): ℜ n →ℜ n with the same property as (1): Step 1: Given x 0 and D 0 = I n , set k = 0 Step 2: Compute F(x k ) Step 3: d − for k = 1, 2,...,n Step 4: If stop else set k = k + 1 and go to Step 2

RESULTS
In order to demonstrate the performance method NDJ, four prominent methods are compared and the comparison was based upon the following criterion: Number of iterations, CPU time in seconds, storage requirement and robustness index. The methods are namely: • NDJ stands for method proposed in this study • The Newton method (CN) • The Fixed Newton method (FN) • The Incomplete Jacobian Newton method (IJN) • MRVF denotes Newton-like method with the modification of right-hand side vector The MRVF was proposed in (Natasa and Zorna, 2001) and IJN proposed by (Hao and Qin, 2008). The stopping criterion used is ||x k+1 -x k ||+||F(x k )||≤10 −8 . We implemented the five methods (CN, FN, MRVF, IJN and NDJ) using MATLAB 7.0. All the calculations were carried out in double precision computer. We introduced the following notations: N: number of iterations and CPU: CPU time in seconds. Problem 1-3 is to show the fitness of our method (NDJ) to small scale and Problem 4-11 are for large scale systems with dense or sparse Jacobian.

DISCUSSION
In Table 10, the robustness index is given by (Natasa and Zorna, 2001): where, t j number of success by method j and n j is the number of problem attempted by method j and the large value of index shows better result and the best possible result is 1.
From Table 1, it can be seen that our proposed method (NDJ) is Cheaper than Newton method (CN), Fixed Newton method (FN) and better than (IJN) and (MRVF). However our method is slower than Newton method (CN) but much faster than Fixed Newton method (FN). In addition CN, MRVF and IJN required to computes and stores the Jacobian in each iteration where as NDJ only vector storage.   Moreover from Table 2-9, our method (NDJ) also uses less computational cost and less CPU time than other four methods (FN, MRVF, CN and IJN), that is why our method (NDJ) is significantly cheaper than CN, MRVF and IJN and also much faster than FN and MRVF, this is more noticeable when the dimension increases. In all the test problem, our method (NDJ) shows a promising result. We can observe that as the dimension of the systems increases, global cost for FN, MRVF, CN and IJN methods increases exponentially where as our method (NDJ) growths linearly. This is because they all require to compute Jacobian matrix in each iterations. Another advantage of our method over FN, MRVF, CN and IJN methods is the storage requirement for the Jacobian matrix to only a vector storage. For instance when n = 1000, CN, MRVF and IJN stores 1000×1000 fully populated matrix which is equivalent to 10 6 memory locations but NDJ stores1000 vectors equivalents to 1000 memory locations only, in general our method has reduced the matrix storage to some vector storage only. These results indicated that NDJ is an improvement; with emphasis on eliminating matrix storage, reducing computational cost and CPU time, as well as avoiding solving n system of equations in each iteration only.

CONCLUSION
In this study, a modification of classical Newton's method for solving system of nonlinear equations is presented. This method (NDJ) approximates the Jacobian into a diagonal matrix, by replacing the derivative computation by a direct function computation. Hence it reduces the computational cost, storage requirements of a matrix, CPU time and avoids solving n linear system in each iteration. The numerical results of the proposed method (NDJ) are very encouraging and it shows that its global cost increases linearly as the dimension of the nonlinear systems increases whereas in CN, FN, MRVF and IJN methods it increases exponentially; furthermore it shows that NDJ can achieve good performance for large nonlinear systems. Therefore to this end we can say that our method (NDJ) is significantly cheaper than the four methods and much faster than FN and MRVF. Finally we can concludes that our method (NDJ) is suitable for small, medium and large scale systems especially when the function derivatives are costly or can't be done precisely.