Analysis of Time-Delayed Neural Networks via Rightmost Eigenvalue Positions

: Neural networks have been frequently used in various areas. In the implementation of the networks, time delays and uncertainty are present and known to lead to complex behaviors, which are hard to predict using classical analysis methods. In this study, stability and robust stability of neural networks considering time delays and parametric uncertainty is studied. For stability analysis, the rightmost eigenvalues (or dominant characteristic roots) are obtained by using an approach based on the Lambert W function. The Lambert W function has already been embedded in various commercial software packages (e


Introduction
During the last several decades, neural networks have received wide interest due to their applications in various areas, such as signal processing, image processing, power systems and optimization (Kim et al., 1996;Haque and Kashtiban, 2005). Because successful performance of some of those systems hinges on stability of neural networks stability analysis and stabilization are significant problems. However, in the implementation of neural networks, due to finite switching speed of amplifiers and signal transmission among neurons, time delays are present and affect stability results. Moreover, designing a network to operate more quickly will increase the relative size of the delay. The time delays lead to complex dynamical behaviors and often induce unstable behaviors. The neural networks that have the time delays are represented by Delay Differential Equations (DDEs). Unlike Ordinary Differential Equations (ODEs), DDEs has an infinite number of roots of the characteristic equations, which are transcendental. For this reason, time-delay systems cannot be handled by using classical methods developed for ordinary systems and thus, are often ignored. Unfortunately, existing approaches for time-delay systems are limited in three critical ways: (1) They approximate time delays in modeling and, thus, reduce accuracy (e.g., Padé approximation); (2) They rely on model-based prediction of future trajectories (e.g., Smith predictor), which is vulnerable to uncertainty; (3) Or, they are dependent upon Lyapunov functions, which induce conservativeness in the results. Those shortcomings mainly come from lack of analytical solutions for timedelay systems. Thus, a new effective approach based on analytical solutions is needed and can contribute advances in theory and development of methodologies.
A systematic approach for time-delay systems has been developed using the Lambert W function 2011). In this study, stability of neural networks is determined based on the Lambert W function-based method. Using the approach, the stability condition can be expressed analytically in term of system parameters. Thus, the approach is more intuitive and similar to methods for non-delay systems. Also, considering uncertainty caused by linearization and/or parametric uncertainty, robust stability analysis is conducted. Conditions for periodic solutions, which can be used to restore various complex patterns (Yu et al., 2008), is presented using the approach. Note that the Lambert W function is already embedded in the various commercial software packages, such as Matlab, Maple and Mathematica. Also, some basic MATLAB codes are available on websites (Yi et al., 2014).
In addition to time delays, in practice, the weight coefficients of the neurons depend on certain resistance and capacitance values which are subject to have uncertainties. Parametric uncertainty, which are frequently neglected in idealized models, can lead to instability not predicted by theory (Marcus and Westervelt, 1989). For example, parametric fluctuation in neural network implementation on Very Large-Scale Integration (VLSI) chips is unavoidable (Cao and Wang, 2005). It is important to ensure the asymptotic stability of the designed network in the presence of such uncertainties (i.e., to ensure the robust stability) (Singh, 2004). Thus, the approach presented in this study will be of interest.

Background
A circuit equation for a network of N neurons (the well-known Hopfield model) was introduced in (Hopfield, 1984). Afterward, considering the finite switching speed of amplifiers, a delay, h', is added to the network model in (Marcus and Westervelt, 1989). The resulting network model is represented by using a system of DDEs given by Equation 1: In the network, u i (t') represents the voltage of i th neuron. The parameters, C i and R i , are the capacitance and resistance of the neuron, respectively. The output of j th neuron is connected to the input of i th neuron through the connection factor, T ij . The transfer function, f(u), is sigmoid. First, consider a network, which consists of identical neurons. Then, after several procedures of linearization and simplification, the equations becomes (Marcus and Westervelt, 1989): where, the gain β is the slope of f i (u) at u = 0 and λ i is the eigenvalues of the connection matrix.
Then, the origin is asymptotically stable when R(s i ) <0 for all i. When R(s i ) >0 for some i, the origin is unstable to perturbations in the direction of the eigenvector associated with s i . Thus, to determine stability of the network, it is essential to obtain the roots of Equation 3. However, because of the exponential term, s h i e , in Equation 3, the characteristic equation is transcendental and the number of roots becomes infinite. The difficulty in using analysis and control methods for ODEs is caused by the fact that it is not feasible to find all the infinite number of roots s i of Equation 3 and to identify the rightmost root among them. The Lambert W function, defined as: Has been known to be useful in solving for the characteristic roots and, subsequently, deriving analytical solutions to DDEs (Corless et al., 1996;Asl and Ulsoy, 2003). In section 3, this Lambert W function-based approach is used to analyze stability and robust stability of neural networks. Also the results are generalized in section 4.

Stability Analysis Using Locations of Roots
As seen in Equation 5, the characteristic roots are expressed in terms of the gain, β and the eigenvalues, λ i and the time-delay, h. Because there are an infinite number of branches of the Lambert W function, W k , an infinite number of roots, s ik , exist. However, the rightmost (i.e., dominant) root, which determines stability, is always obtained by using the principal branch, k = 0 ( Fig. 1) (Shinozaki and Mori, 2006). This is the primary advantage of the approach. Thus, the stability condition is given in an analytical form derived from the solution in Equation 5. That is, the system is stable if and only if: Then, for stable networks, bifurcation happens when the rightmost root cross the imaginary axis. That is, R(s i0 ) = 0 and ∂R{s i0 }/∂γ>0, where γ is a parameter of the system. That is: Figure 1 show Spectrum of roots for λ 1 = 1. Due to the time-delay, the number of the roots of the characteristic equation becomes infinite, unlike nondelayed systems. Using the branch of the Lambert W function, all the roots in spectrum are obtained and each root is distinguished individually. Moreover, the principal branch of the Lambert W function identifies the rightmost root, which determines stability of the network, among those roots.
For the second condition in Equation 7, the analytical expression for the derivative of the principal branch of the Lambert W function, which is given by Equation 8:
Thus, this network is stable. Figure 1 shows the spectrum of roots for λ 1 = 1. Due to the time-delay, the number of the roots of the characteristic equation becomes infinite, unlike non-delayed systems. Using the branch of the Lambert W function, all the roots in the spectrum are obtained and each root is distinguished individually. Moreover, the principal branch of the Lambert W function identifies the rightmost root, which determines stability of the network, among those roots. As seen in Fig. 1, to determine stability, one does not have to calculate all the roots. Just with one root in the spectrum, stability is determined efficiently. According to the criterion in Equation 6, this network is stable. On the other hand, the rightmost root also for λ 1 = 1 but with for σ = 8 (thus, β = 2) is s 10 = 0.3748. Because the dominant root is in the Right Half Plane (RHP) the network becomes unstable. Figure 2 shows the spectrum of the characteristic roots.

Example 2: Complex Coefficients
Since the Lambert W function is defined also for complex arguments, the approach can be readily applied to some other types of neural networks, for example, introduced in (Gopalsamy and Leung, 1996;Olien and Belair, 1997) and the references therein. Consider a network from (Gopalsamy and Leung, 1996) After linearization about the origin, one gets Equation 13: Because the two coefficient matrices in Equation 13 commute and, thus, are simultaneously triangularizable (Radjavi and Rosenthal, 2000). If c 1 c 2 >0, the system is decoupled into two scalar equations of complex coefficients as Equation 14: For example, for b = 1 and c 1 = c 2 = 2, one gets: Even for complex arguments, the stability condition in Equation 6 holds (Shinozaki and Mori, 2006). Thus, the stability of the network is analyzed using the Lambert W function-based approach. In (Gopalsamy and Leung, 1996), the condition for stability is obtained via bifurcation analysis. Instead, the condition is given in terms of the Lambert W function. That is, the network is stable if and only if ( ) Figure 3 shows the values of the real parts of the rightmost roots of Equation 14, for b = 1 and c 1 = c 2 = 2. As the time-delay increases, the real part also increases.
When h is 0.302, bifurcation happens and the network becomes destabilized, which agrees with the results in (Gopalsamy and Leung, 1996). This approach can be extended to robust stability of complex-valued neural networks, which have been found highly useful in broadening the scope of applications of neural networks (Hirose, 2009). This will be discussed later more in details in Subsection 3.3.
Stability problems of neural networks having time delays have been studied in literature using bifurcation analysis (Yu et al., 2008;Marcus and Westervelt, 1989), Also, robust stability has been studied in literature using Linear Matrix Inequalities (LMIs) (Cao and Wang, 2005;Zhang et al., 2005;Li et al., 2010) and Lyapunov functions (Yucel and Arik, 2009). The LMI-based techniques have been successfully used to tackle various stability problems for neural networks with time delays (Zhang et al., 2005). Compared to such methods, the method based on the Lambert W function directly finds the roots of the characteristic equations, which have infinite number of roots. Thus, it provides stability conditions analytically expressed in terms of system parameters and problems are relatively easy to formulate. It provides sufficient and necessary conditions for stability and, thus, conservativeness of the methods based on Lyapunov functions can be reduced. Also, one can determine how stable the system is as well as whether stable or not, from the rightmost characteristic roots. Therefore, it is possible to address stability problems in a way similar to non-delay problems.

Existence of Oscillations
To increase information that can be stored in networks, research regarding periodic solutions has been conducted extensively (Yu et al., 2008) and the references therein). When the characteristic roots cross the imaginary axis, conditions for existence of purely imaginary roots (i.e., R( srightmost ) = 0 and I( srightmost ) ≠ 0) can be used to study bifurcation of time-delay systems. Bifurcation of dynamical systems is investigated by using the normal form method and center manifold theorem in literature (Stépán, 1989). Alternatively, such analysis is conducted by using graphical approach (i.e., Nyquist stability criterion) (Moiola and Chen, 1996) and was applied to neural networks (Yu and Cao, 2007). Here, the condition for existence of non-zero imaginary parts is expressed in terms of the Lambert W function. As seen in Fig. 4, if the real argument of the Lambert W function, H, is equal to or greater than -1/e, the value of W 0 (H) is real and the network does not show oscillation. However, if H is smaller than -1/e or complex, the rightmost roots have non-zero imaginary parts and the trajectory shows oscillation. Thus, one can conclude that the network system has a periodic solution if and only if the parameters satisfy the two conditions (i.e., 1) R( srightmost ) = 0 and 2) I( srightmost ) ≠ 0) simultaneously: Which are derived from the roots in Equation 5.

Example 3: Antiferromagnetic Network
The antiferromagnetic connection matrix given by Equation 18 (Marcus and Westervelt, 1989): The eigenvalues of the matrix are Equation 19: (thus, σ = 9.047). That is, s rightmost = ±2.0287j. Figure 6 shows oscillatory trajectories (x i ) of the networks for β = 2.2617, h = 1 and N = 4. Because the rightmost characteristic roots are purely imaginary, the amplitude of the oscillation sustains constantly over time.

Robust Stability
As discussed in Introduction, when uncertainty exist in time-delay systems, robust stability is one of the primary concerns for control. Considerable amount of results on the topics has been presented in literature. Mostly, robust stability has been studied based on the Lyapunov-Krasovskii functional method. The approach yields stability conditions given by Linear Matrix Inequality (LMI) (Singh, 2004). Instead of constructing Lyapunov-Krasovskii functionals and deriving LMI conditions, robust stability also can be addressed directly from the locations of the rightmost characteristic roots of delay differential equations (Shinozaki and Mori, 2006; and references therein). For example, assume that β has uncertainty as β + ∆β and ∆β = ±0.2β. Then, using the results in (Shinozaki and Mori, 2006), without exhaustive search for every value in β-0.2β≤β≤β +0.2β, just with one root, s 0 = 1/h×W 0 ((β +0.2β)λ i he h ) -1, robust stability of the network can be determined.
Besides neural networks of the real parameters, recently, the introduction of complex-valued neural networks, which handle more information by using complex-valued parameters and variables, has widened applications of artificial neural net-works (Garimella, 2006), such as digital signal processing, Magnetic Resonance Imaging (MRI) reconstruction (Hirose, 2009). As shown in this section, the Lambert W function can be used to analyze dynamics of such complex-valued neural networks.

Solution to General Systems of DDEs
The method for first-order scalar DDEs based has been extended to general systems of DDEs using the matrix Lambert W function as: where, x(t)∈ Rn is a state ∈vector; A∈ Rn×n, A d ∈ Rn×n. In , the solution, which is expressed in terms of the matrix Lambert W function, to (20) was developed and given by: For detailed explanation regarding solving for Q k and calculating C I k from initial conditions, refer to . In a way to similar to scalar DDEs in Sect. 3., from the solution form in Equation 21, stability analysis can be conducted.

Rightmost Eigenvalues and Stability
For systems of DDEs, stability is determined in a similar way to the scalar case. That is, the finite (n) eigenvalues of S 0 , among all the S k , have the rightmost one, which determines the stability of the system . For scalar cases, this has been proven in (Shinozaki and Mori, 2006). Such a proof can readily be extended to systems of DDEs where A and A d commute.
Although such a proof is not currently available in the case of the general matrix-vector DDEs, the same behavior has been observed in all cases where Ad does not have repeated zero eigenvalues. Refer to Shinozaki and Mori, 2006) and references therein for more details about stability analysis using the Lambert W function.

Example 5: Multi-Neuron Network
Consider a network from (Yu and Cao, 2007): Figure 7 shows the rightmost eigenvalues of the system (22) after linearization. As seen in Fig. 7, bifurcation occurs between h = 0.5182 and h = 0.5183, which agrees with the results in (Yu and Cao, 2007). Stability was investigated by using bifurcation method and applying the Nyquist stability criterion in (Yu and Cao, 2007). Compared to such methods, the Lambert W function-based approach enables one to tell how stable the system is from the exact positions of the rightmost roots. If the rightmost roots of a system are located further from the imaginary axis, the system is more stable and vice versa.

Stability Radius and Robust Stability
The decision on robust stability can be made by using robust stability indices (Hu and Davison, 2003). Assume that the perturbed system (20) can be written in the form: where, E ∈ R n×m , F i ∈R li×n and ∆ i ∈ R m×li denotes the perturbation matrix. Provided that the unperturbed system (20) is stable, the real structured stability radius of Equation 23 is defined as (Hu and Davison, 2003): where, ∆ = [∆ 1 ∆ 2 ] and σ 1 (∆) denotes the largest singular value of ∆. The largest singular value, σ 1 (∆), is equal to the operator norm of ∆, which measures the size of ∆ by how much it lengthens vectors in the worst case. Thus, the stability radius in Equation 24 represents the size of the smallest perturbations in parameters, which can cause instability of a system. The real stability radius problem concerns the computation of the real stability radius when the nominal system is known. The stability radius is computed by using the method presented in (Hu and Davison, 2003). The obtained stability radius provides a basis for assigning eigenvalues for robust stability of systems of DDEs with uncertain parameters. For example, consider the network in Equation 22 and the rightmost characteristic roots in Fig. 7. Figure 8 shows computed stability radius for each stable roots in Fig. 7. For h = 0.5180, if there is no uncertainty in coefficients, the network is stable. However, the corresponding stability radius is r R = 0.0101. Thus, if the size of the uncertainty is larger than the radius (i.e., σ 1 (∆) >0.0101), the system can be destabilized by the uncertainty. On the other hand, if σ 1 (∆) < 0.0101, the system with uncertainty is robustly stable. for different values of time delay, h. The rightmost roots are identified using the novel property of the Lambert W function and used for stability analysis. Also, the distance from the imaginary axis of stable roots informs how stable the system is  Fig. 7. For h = 0.5180, the corresponding stability radius is r R = 0.0101. Thus, if the size of the uncertainty is larger than the radius (i.e., σ 1 (∆) >0.0101), the system can be destabilized by the uncertainty. That is, the system is not robustly stable

Conclusion
Stability and robust stability of neural networks has been investigated through solving for the characteristic roots of delay differential equations. Although DDEs render transcendental characteristic equations, the characteristic roots and dominant ones can be found by using the Lambert W function. Based on the Lambert W function-based approach, stability conditions are derived from the locations of the characteristic roots in the complex plain. Besides stability, from the locations of the roots (i.e., distance from the imaginary axis), it is possible to know how stable the systems is. Also, using the property of the Lambert W function, robust stability conditions are given in terms of parameters without constructing Lyapunov functions.
In future, the Lambert W function-based approach can also be applied to more general complex-valued neural networks. Also, stabilization of neural networks can be investigated further using the presented approach based on the Lambert W function. Applications to physical systems (e.g., for pattern recognition) are being studied by authors.