A Modification of the Ridge Type Regression Estimators

Problem statement: Many regression estimators have been used to remedy multicollinearity problem. The ridge estimator has been the most popular one. However, the obtained estimate is biased. Approach: In this stuyd, we introduce an alternative shrinkage estimator, called modified unbiased ridge (MUR) estimator for coping with multicollinearity problem. This estimator is obtained from Unbiased Ridge Regression (URR) in the same way that Ordinary Ridge Regression (ORR) is obtained from Ordinary Least Squares (OLS). Properties of MUR estimator are derived. Results: The empirical study indicated that the MUR estimator is more efficient and more reliable than other estimators based on Matrix Mean Squared Error (MMSE).Conclusion: In order to solve the multicollinearity problem, the MUR estimator was recommended.


INTRODUCTION
Consider the following linear regression model: . If X'X is singular or near singular, we say that there is multicollinearity in the data. As a consequence, the variances of elements of LŜ β are inflated. Hence, alternative estimation methods have been proposed to eliminate inflation in the variances of LŜ β . Hoerl and Kennard (1970) proposed Ordinary Ridge Regression (ORR) as: Usually 0 < K < 1. This estimator is biased but reduces the variances of the regression coefficients. Subsequently, several other biased estimators of β have been proposed (Swindel, 1976;Sarkar, 1996;Batah and Gore, 2008;Batah et al., 2009;Arayesh and Hosseini, 2010;Asekunowo et al., 2010;Hirun and Sirisoponsilp, 2010;Rana et al., 2009). Swinded (1976) where, b is a prior estimate of β. As K increases indefinitely, the MRR estimator approaches b. Crouse et al. (1995) defined the Unbiased Ridge Regression (URR) estimator as follows: for K>0. They also proposed the following estimator of the ridge parameter K: with Z = XT, L = T'β where Z'Z = T'X'XT = A= diag (λ 1 , λ 2 , ….., λ P ). The diagonal elements of A are the eigenvalues of X'X and T consists of corresponding the eigenvalues of X'X. Hence OLS, ORR and URR of L are written as In this study, we introduce an alternative shrinkage estimator , called Modified Unbiased Ridge (MUR) estimator. This estimator is obtained from URR in the same way that ORR is obtained from OLS. It is observed that OLS is unbiased but has inflated variances under multicollnearity. Similarly, URR suffers from inflated variances while eliminating the bias. The construction of MUR is based on the logic that just as ORR avoids inflating the variances at the cost of bias, MUR would have similar properties. With pre-multiple the matrix [I -K (X'X + KI p ) −1 ] to reduce the inflated variances in OLS, so that we expect the same effect with URR. so that we expect the same effect with URR. This is our motivating the alternative modified estimator. In this respect, the main object of this paper is that the MUR estimator performs well under the conditions of multicollinearity. The properties of this alternative modified estimator are studied , and some conditions for this estimator to have smaller MMSE than ORR and URR are derived also. In addition, as the value of K must be specified for K in MUR in the same way as in ORR and URR, so three different ways for determining K are compared using simulated data.

MATERIALS AND METHODS
The proposed estimator: We propose the following estimator of β: where, J~ N( where,

Matrix Mean Squared Error (MMSE):
where, tr denotes the trace. Then: where, {λ i } are eigenvalues of X'X.
Comparison with other estimators: MUR is biased and it is therefore compared with other estimators in terms of MMSE. We obtain conditions for MUR to have smaller MMSE than another estimator.

Comparison with ORR:
The MMSE of ORR is (Ozkale and Kaçiranlar, 2007): so that: Consider: and hence: Then: From (11): Since the matrix (Farebrother, 1976), Φ is n.n.d. if and only if: Hence we have the following result.
Result 2: MUR has smaller MMSE than URR if: The condition of result (2) is verified by testing: β≤σ . Under the assumption of normality: and the test statistics: under H 0 . The conclusion is that MUR has a smaller MMSE than URR if H 0 is accepted and hence Result (2) holds.
Optimal ridge parameter: Since the MMSE of MUR depends on the ridge parameter k, the choice of k is crucial for the performance of MUR. Hence we find conditions on the values of k for MUR to be better than other estimators in terms of SMSE.
Result 3: We have: Proof: Result (3) can be proved by showing that: which is obtained from (12) and (18). This completes the proof. Next, we compare SMSE of Ĵ (L (k)) with that of OLS component-wise. Notice that the MUR estimator reduced to OLS when k = 0. The i-th component for SMSE of L of OLS is given by: We state the following result.
Proof: Result (4) can be proved by showing that: which is obtained from (12) and (22). This completes the proof. Furthermore, differentiating i Ĵ SMSE (L (k)) with respect to k i and equating to zero, we have the following equation: Thus, the optimal value of the ridge parameter k i is: From (21), (23) and (24), it can be easily verified that k i1 <k i(FG) <k i2 if 2 2 i i L 0 λ − σ > . In case k = k 1 = k 2 … = k p , we can obtain k as the harmonic mean of k i(FG) in (24). It is given by: Using an argument from Hoerl et al. (1981), it is reasonable to adopt the harmonic mean of the regression coefficients. Note that k (FG) in (25) depends on unknown parameters L and σ 2 and hence has to be estimated.
Estimating the ridge parameter k: In this section, we propose to construct MUR by using the operational ridge parameter proposed by Hoerl et al. (1981) and Crouse et al. (1995). First, since the harmonic mean of optimal ridge parameter values, (see (24)) depends on the unknown parameters L and σ 2 , we use their OLS estimates. The operational ridge parameter in (25) This is called the (FG) ridge parameter. Second, the HKB ridge parameter (Hoerl et al., 1981) Third, CJH ridge parameter (Crouse et al., 1995) is: Using these three operational ridge parameters, we compare the following ten estimators: • OLS • ORR using the HKB ridge parameter (ORR (HKB)) • ORR using the CJH ridge parameter (ORR (CJH)) • ORR using the FG ridge parameter (ORR (FG)) • URR using the HKB ridge parameter (URR (HKB)) • URR using the CJH ridge parameter (URR (CJH)) • URR using the FG ridge parameter (URR (FG)) • MUR using the HKB ridge parameter (MUR (HKB)) • MUR using the CJH ridge parameter (MUR (CJH)) • MUR using the FG ridge parameter (MUR (FG))
The data was then used by Course et al. (1995) to compare SMSE performance of URR, ORR and OLS. Recently, Batah et al. (2009) used the same data to illustrate the comparisons among OLS and various ridge type estimators. We now use this data to illustrate the performance of the MUR estimator to the OLS, ORR and URR estimators to compare the MMSE performance of these estimators. show that the magnitude of shrinkage is not enough.

DISCUSSION
When biased and unbiased estimators are available, we prefer unbiased estimator. Crouse et al. (1995) (2) holds. That is, MUR has smaller MMSE than URR.

CONCLUSION
In this study article we have introduced Modified Unbiased Ridge (MUR) estimator. Comparison of this estimator to that ORR and URR has been studied using the MMSE. Conditions for this estimator to have smaller MMSE than other estimators are established. The theortical results indicate that MUR is not always better than other estimators in terms of MMSE. MUR is best and depends on the unknown parameters β, σ 2 and also using the ridge parameter k. for suitable estimates of these parameters, MUR estimator might be considered as one of the good estimators using MMSE.