PERFORMANCE OF RIDGE REGRESSION ESTIMATOR METHODS ON SMALL SAMPLE SIZE BY VARYING CORRELATION COEFFICIENTS: A SIMULATION STUDY

When independent variables have high linear correla tion in a multiple linear regression model, we can h ve wrong analysis. It happens if we do the multiple li near regression analysis based on common Ordinary L east Squares (OLS) method. In this situation, we are sug gested to use ridge regression estimator. We conduc t some simulation study to compare the performance of ridg e regression estimator and the OLS. We found that H oerl and Kennard ridge regression estimation method has better performance than the other approaches.


INTRODUCTION
Main goal of multiple linear regression model is to determine the best set of parameters, β i , so that the predicted value of dependent variables close to the real values (Orlov, 1996). In multiple linear regression models, we normally assume that the independent variables are independent. However, in practice, the explanatory variables may be correlated between each other. This inter-relation between the explanatory variables is called multicollinearity. It is the undesirable situation and can happen when the correlations among the independent variables are strong. It has several effects as has been described by Judge et al. (1988). One of them is that it increases the standard errors of the coefficients. In this situation, the independent assumptions are no longer valid in multiple linear regression models. The regression coefficients which are based on Ordinary Least Square estimator (OLS) tends to become unstable in the presence of multicollinearity. Wethrill (1986) also mentioned that multicollinearity is a serious problem when we make inferences for a model so that it must be handled appropriately.
Except due to strong natural linear correlation between independent variables, multicollinearity can happen due to the present of high leverage points. Recent researches focused on high leverage points correspond to outlier are Bagheri and Midi (2009).
Ridge regression estimator methods have been proposed as alternatives to the OLS estimators when the independent assumption has not been satisfied in the analysis. Several methods have been proposed for estimating the ridge parameter k and consider a criterion for comparison of the estimators. We present several methods based of ridge regression estimators.
Hence, ridge regression estimator has been proposed as an alternative to the OLS estimators when the independent assumption has not been satisfied in the analysis. The ridge estimator constrains the length of the regression coefficient of the estimator in the presence of multicollinearity. Ridge regression will be able to minimize the variance of the estimators when the design matrix is not invertible. The modification of design matrix to make its determinant different form 0 causes the estimator to be biased. This method is significantly reduces the variance of the estimators. Through this research, we want to observe how are the parameters of ridge regression estimator in different level of correlation coefficients by using Monte Carlo procedure.

Ridge Regression Model
Multicollinearity refers to a situation in which two or more predictor variables in a multiple regression model are highly correlated. Multicollinearity occurs when there is a linear relationship between one or more of the independent variables. In this situation, the regression coefficients change significantly in response to small changes in the model. The regression coefficients cannot be estimated with great accuracy because the coefficients possess large variance.
The ridge regression estimator is much more stable than the OLS estimator in the presence of multicollinearity. The ridge estimator restricts the length of the coefficients estimator in order to reduce the effects of multicollinearity (Hocking et al., 1976). In the presence of multicollinearity, Hoerl and Kennard (1970) introduced the ridge estimator as an alternative to the OLS estimator when the independent assumption is not longer valid. The ridge estimator is shown as follow Equation (1) (Hoerl and Kennard, 1970): where, the I denotes an identity matrix and k is known as ridge parameter. The MSE of k β is shown as follow: (2) depends on unknown parameters k, β and σ 2 , which can't be calculated in practice. As k increase from zero to infinity, the regression estimates will approximately equal to zero. It yields minimum ( ) k MSE β compared to the OLS estimator, although these estimator results in bias, for a certain value of k (Hoerl and Kennard, 1970). In practice, we have to estimate k from the real data instead.
Standard model of a multiple linear regression can be expressed into canonical form. An orthogonal matrix D exists such that: where, C = X'X and Λ = diag (λ 1 , λ 2 ,…, λ p ) contains the eigenvalues of the matrix C, then the canonical form of the model (1) is Equation (3): where, X* = XD and α = D'β. The general form of OLS estimator is shown as follows Equation (4): Then, the ridge estimator is written as Equation (5): where, K = diag (k 1 , k 2 ,…,k p ), k i >0. The ridge estimator in Equation (4) is known as general form of ridge regression (Hoerl and Kennard, 1970). According to Hoerl and Kennard (1970), the value of k 1 which minimizes the Equation (6): Is: where, σ 2 denotes the error variance of model Equation (1), α i is the ith element of α. Equation (7) shows that the values of k i fully depends on the unknown σ 2 and α i . Since σ 2 and α i are unknown, these values must be estimated from the observed data. Bhar and Gupta (2001) proposed a new criterion of detecting outlier in experimental designs which is based on average Cookstatistic. Meanwhile, Zhou and Zhu (2003) realized the fact that in practice, experiments may yield unusual observations (outliers). In the presence of outliers in a data, estimation methods such as ANOVA, truncated ANOVA, Maximum Likelihood (ML) and modified ML do not perform well, since these estimates are greatly influenced by outlier. Zhou and Zhu (2003) verified that with robust designs, one can get efficient and reliable estimates for variance components regardless of outliers which may happen in an experiment. Then Goupy (2006) conducted further research regarding outlier in an experiment who described how to discover an outlier and estimate its true value and recently, Fitrianto and Midi (2013) who compared classical and robust approach in experimental design. The method is based on the use of a dynamic variable and the ''small effects'' of the Daniel's diagram.

Method for Estimating Ridge Regression Parameters
Several methods have been proposed in order to define a new estimator that can perform better compared to the existing methods. In this part, we present some methods for estimating ridge parameter k. Hoerl and Kennard (1970) found that the best method to estimate α(k) ⌢ is to use k i = k for all i and they suggested k is to be HK k ⌢ (or HK) where Equation (8): If σ 2 and α are known, then HK k ⌢ is sufficient to give ridge estimators having smaller MSE than the OLS estimator. Hocking et al. (1976) defined a new method for choosing parameter k. They suggested an estimator of k by using HSL k ⌢ (or HSL), which produces the following estimator Equation (9): Recently, Alkhamisi and Shukur (2007) suggested a new approach for choosing the ridge parameters k by adding 1/λ max to some well-known estimators, where λ max is the largest eigenvalues of X'X. They applied the modification to the previous estimator which was proposed by Hocking et al. (1976) in order to define a new estimator NHSL k ⌢ (or NHSL) Equation (10)

The use of Monte Carlo Simulation
Monte Carlo method is a stochastic technique which is used to investigate problems based on the use of random numbers and the probability statistics. We can use Monte Carlo method to solve physical problems, for example it allows us to examine more complex systems. With Monte Carlo method, we can sample the large system in a number of random configurations. Bagheri and Midi (2009) also conducted Monte Carlo simulation study in a robust approach in the presence of multicollinearity.
Simulation study will be discussed to compare the performance of ridge estimators under several degrees of multicollinearity. Different ridge estimators corresponding to different values of ridge parameter k are considered. McDonald and Galarneau (1975) and several other researchers used the following equation to generate the explanatory variables Equation (11) where, z ij is independent standard normal pseudo-random numbers and γ is linear correlation between any two explanatory variables. The n observations for the dependent variable y are determined by Equation (12): where, ε i are independent normal (0,σ 2 ) pseudo-numbers. The comparison is based on the MSE criteria. The MSEs of ridge estimators are evaluated by Equation (7).

Simulation Design
The simulation is conducted by using SAS release 9.2. To achieve different degrees of correlation, the explanatory variables were generated using the Equation (11). Size of sample to be considered in this research is small sample of size 20 with number of explanatory variables of equal to 10. Different values of correlations are considered in the simulation study are 0.5, 0.7 and 0.9. These three values of to represent low, moderate and high correlations between explanatory variables. The explanatory variables need to be standardized so that they will be in correlation form. Meanwhile, five different values of standard deviation to be considered in this study, which are 0.1, 0.5, 1.0, 5.0 and 10.0.

Performance Measures of the Estimators
For given values of p, σ and γ, we repeated the experiment by 1000 times. For each replication, r = 1,2,3,..,1000, the values of these three ridge estimators and the corresponding parameters, k will be estimated using the standardized variables and then the estimated coefficients are transformed back to the original model. The k values were computed based on its corresponding method.
The performance of the estimators is evaluated in terms of the averaged mean square error (Dorugade and Kashid, 2010) with the following equation: The comparison between estimated MSEs are then based on the values of p, σ and γ.

RESULTS AND DISCUSSION
Results of the estimators performance are displayed in

JMSS
The MSEs of OLS estimator are lower than the MSEs of NHSL estimator for all levels of correlation when the value of σ = 0.1. However, the NHSL estimator performs better than OLS estimator when σ>0.1. From Table 1, we can conclude that HK estimator performs better than the OLS and other ridge estimators. We display graphical plot of MSE (of each ridge regression method) versus the level of correlation coefficients by different values of standard deviations on Fig. 1. We compare the MSEs of each estimator graphically by varying the standard deviations and the correlations between the explanatory variables. The results of different levels of correlations corresponding to standard deviation as shown below.
In Fig. 1 we can see that when the data has small variability (which are represented by σ = 0.1 and σ = 0.5) the MSE values between ridge regression method are about the same so that the small differences between them can be neglected. But when the value of standard deviation is at least one, we can observe that the MSE values increases as the standard deviation increases, regardless the methods. Moreover, we can see that the OLS estimator has the highest MSE compare to ridge estimators and within the regression estimation methods, the HK estimator performs better than the HSL and NHSL estimators for all levels of correlations.

CONCLUSION
In this article, we did simulation studies of several methods for estimating the ridge parameters. The performance of each ridge estimator depend on the standard deviation (σ) and the correlations between of explanatory variables (γ 2 ). For σ = 0.1, HK estimator and HSL estimator have smaller MSE than the OLS estimator for all levels of correlations. However, the OLS estimator is reasonably better than NHSL estimator for all levels of correlations for this given value of . HK estimator might be recommended to be used to estimate the ridge parameter k. Further investigation of ridge estimators is needed in future in order to make any definite statement.

ACKNOWLEDGEMENT
The reachers are grateful to University Grants Scheme by Universiti Putra Malaysia for awarding a research grant for supporting the research.