A New Method for Accuracy Assessment of Sub-Pixel Classification Results

: Problem statement: Classification accuracy assessment is the main stage of information extraction to evaluate the performance of a classifier. According to the output's classifiers (thematic map/fraction map), a properly strategy for accuracy assessment should be taken into consideration. Since pure pixels are used in traditional accuracy assessment of full pixel classifiers they are not suitable for assessment of sub-pixel classifiers. The objectives of this study were to find a standard sub-pixel accuracy assessment method for evaluation of the sub-pixel classifiers. For this purpose many efforts had been taken and recently some methods and measures such as entropy and cross-entropy had been proposed for sub-pixel accuracy assessment. These methods had their own shortcomings which seriously a fuzzy ground truth data set was needed, the matter that is not available simply. Approach: In this study recently sub-pixel classifier accuracy assessment methods were explored and a new method based on correctness coefficient parameter for the sub-pixel accuracy assessment was introduced. In order to evaluate the CC method, a sub-image of the AVIRIS of hyper spectral data was taken over an agricultural area of California, USA in 1994. The study area consisted of 16 classes. Sub-pixel accuracy assessment methods were discussed. The experiment results using AVIRIS data demonstrated the ability of the new accuracy assessment method. Results: Indeed, in proposed method the matching rate of fraction maps with ground truth data was quantified as correctness coefficient parameters. As a result, flexibility and consistency of sub-pixel accuracy assessment certified by correctness coefficient regarding the type of available data and classification methods. The obtained overall CC over LSU method using 120 bands is about 84.9%. In contrast, the obtained results in terms of OA and Kappa coefficient over LSU method which were achieved by maximum value rule on the fraction maps are 86 and 84% respectively. The Kappa coefficient value is close to overall CC of LSU method. Conclusion: Hence, evaluation and experiments demonstrated that the CC method as an accuracy assessment parameter of a soft classifier can be substituted reasonably by traditional accuracy parameters.


INTRODUCTION
Remote sensing is an attractive source of data for land cover mapping applications. Classification traditionally is defined as a mapping function from the image apace into a nominal space which each pixel has one label. Usually the result of the classification is a thematic map which each pixel is allocated to a specific class. In some cases classification tries to delineate objects on the real world and this is done by preprocessing the image (e.g., segmentation). In the other hand, some of the classification methods defines the per pixel fraction of each class and allow calculating the correct area estimation of the classes.
There are many classifiers that so far have been used and some of them exist in the state of the art soft wares like maximum likelihood (MLC), minimum distance and so on. Traditional classifiers often attempt to generate a thematic map but this leads to the incomplete area estimation of the classes. Because of the mismatching of the sensor grid and the real object boundaries, some mixed pixels (mixels) will appear in the image [4] .
The grey value of such mixels is a composition of the radiometric properties of the several classes (objects) and therefore generates some confusion in classification procedures [14,15] . Traditional classifiers like MLC assign each pixel to only one class. Consequently the mixels usually are labelled erroneously. Such classifiers are inappropriate for mixels which contain two or more land cover classes. Hence, a sub-pixel classifier is required. After each classification the classified image must be investigated and its accuracy should be reported. In respect of the result type (Thematic map, fraction map...) we can choose an adequate strategy for accuracy assessment. At last, some parameters, tables and maps will be calculated and generated to show the accuracy of the result.
In this study we try to show some aspects of the sub-pixel accuracy assessment of the classified maps resulted from the sub-pixel classifiers. However a new method and a rectified version of traditional accuracy measures are proposed for using in accuracy assessment of the sub-pixel classifiers.

Sub-pixel classification methods:
The main problem and limitation of traditional hard (pixel based) image classification procedures is in the classification of mixed pixels. Mixed pixel classification is a process which tries to extract the proportions of the pure components of each mixed pixel. Sub-pixel classifications have been used to represent land cover when pixels may have multiple and partial class membership. To resolve the mixed pixel problem, there are different approaches. Some of the most important soft classification methods are: (i) Deterministic approaches; (ii) Fuzzy set theory based approaches [13] ; (iii) Neural network based approaches [12,18] ; (iv) Linear mixture modelling approach [3,11,17] . Among of these approaches we chose the linear mixture modelling approach to produce some (semi) fuzzy results and use them to test the accuracy assessment approaches.
There are two different mixture models for mixed pixel classification [3] : The nonlinear mixing and the linear mixture model. The nonlinear mixture model for unmixing analysis considers not only the pixel of interest, but also involving the neighbouring pixels i.e., each photon that reaches the sensor has interacted with multiple scattering between the different class types. In the linear mixture model, each pixel is modelled as a linear combination of a number of pure materials or endmembers. These mixture models are shown in the Fig. 1 and 2. The linear mixture model is known as the spectral unmixing. Spectral unmixing is a method which the user allowed to determine information on a sub-pixel level and to study decomposition of mixed pixels [9] . In this research, the linear mixture model is concerned and linear unmixing model is used to classify a hyperspectral image. This linear mixture model can be mathematically described as a set of linear vectormatrix equations: Where: i = 1,…,m (m is the number of bands) j = 1,…,n (n is the number of endmembers or classes) DN i = The i×1 hyperspectral vector of each pixel E ij = The j×i endmembers spectrum matrix and it is the weighting fraction of each endmember. Each column in E ij matrix is the spectrum of one endmember f j = The j×1 fraction vector of each endmember for the pixel and ε i is the error term of mathematical model This model can be described as in the matrix form: Solving the Eq. 1 results the unconstrained unmixing using no constrain. The resulting fractions may have negative values and are not constrained to sum to unity. In order to avoid this, the sum to unit constraint is added to the equations of the unmixing process. Applying the condition that all the resulting fractions must sum to unit is referred to partially constrain unmixing. However, fraction values which are negative or greater than one are still possible [16] . Fully constrained unmixing implies an additional condition in that all determined endmember fractions must be between 0-1. It should be noted that the final results of unmixing algorithm depend to the type and number of endmembers. Therefore, any changes applied to the reference endmembers will cause changes on the fraction map results [3] .
A solution for the linear unmixing problem requires that; the sum of the coefficients equals one, because ensure the whole pixel area is represented in the model and also each of the fraction coefficients be nonnegative to avoid negative sub-pixel areas. The first requirement can be modelled by a constraint equation, for the second requirement, the coefficients need to be constrained by: Together, the mixing equations and the constraints describe a model that must be solved for each pixel which should be decomposed, i.e., given DN and E, we have to determine f and ε in Eq. 1 [9] . We will have some fraction maps (f) which hopefully shows the real fraction of each class for pixels as the output.
Accuracy assessment methods of the classifiers: Pixel based accuracy assessment methods: Accuracy assessment is an essential post classification stage. Accuracy of the results is expressed in various forms relative to the classification results and method. The result of the common classification methods (e.g., MLC) is in the form of land cover/use map and usually the accuracy of it is assessed by comparing it to a ground truth map. The ground truth or reference map is usually stored in the digital form and defines well known land cover types for some pixels of the scene. Pixel by pixel comparison of these two maps results an error (or confusion) matrix.
From the error matrix some error and accuracy measurements are derived which each of them show some error or accuracy aspects of the final results. One of the most popular parameter calculated on the basis of the error matrix is overall accuracy. This parameter equals the ratio of sum of the diagonal elements of the error matrix on the number of pixels which have been correctly classified. For each category (class), an accuracy parameter is also defined. In each row, the ratio of the diagonal component (for each class) on the sum of pixels of that row is called user's accuracy. Analogously this ratio is calculated for each column and is called producer's accuracy.
Based on the error matrix another measure for accuracy is defined which is called Kappa coefficient. This accuracy criterion is calculated by Richards [10] : In which: N = Total number of test pixels i ij j x x + = ∑ (i.e., sum of the elements for row i)

sum of the elements for column j)
Commission error is defined as the ratio of the sum of the off diagonal components in each row to the number of pixels of that row. Omission is a similar error measurement for columns. Hence for each row we can calculate commission and user's accuracy and for each column omission and producer's accuracy are calculated. These factors need ground truth data and comparing thematic map and ground truth which results an error matrix. As Congleton and Green [2] have mentioned, error matrix is recommended only for identifying sources of confusion (i.e., differences between the remotely sensed map and the reference data) and not just error in the remotely sensed classification and this fact would be considered more important in the sub-pixel accuracy assessment.
Also, generating a thematic map in sub-pixel classification results are not straight forward and some other post-processing (e.g., thresholding) must be applied. For this reason, accuracy assessment of the sub-pixel classification results is not similar to the common accuracy assessment methods. If we want to use the traditional accuracy assessment (e.g., confusion matrix) we have to generate a thematic map and then compare it with a ground truth map.

Sub-pixel accuracy assessment methods:
In respect to the result's type (thematic map/fraction map), an adequate strategy for accuracy assessment of the classification results must be chosen. Traditional methods of accuracy assessment for conventional pixelbased classification are not perfect for sub-pixel classification accuracy assessment. Because, training data and ground truth are pixel-based and we can not use directly any pixel based method for accuracy assessment of the sub-pixel classification results. Although in some cases the only way to compute the accuracy of a sub-pixel classifier is to harden its [5] .
On the basis of the sub-pixel classification results, some methods have been proposed to estimate the accuracy of such a classification method. Foody [5] has an excellent review on the available sub-pixel accuracy assessment techniques. The most of the methods that he mentioned need to a fuzzy ground truth map the matter that is not available simply [2] . Avoiding the dependency to the ground truth data, entropy is defined. Entropy measures the uncertainty in a single value of a statistical variable and is defined as the information content of a piece of information that would reveal this value with perfect accuracy. This quantity is weighted by the probability that value occurs and summed overall values, which gives [7] : In which: N = Number of classes P(x i /c i ) = The posteriori probability of the class C i in the pixel x p Thus entropy is calculated per pixel.
The entropy parameter has several limits and disadvantages. One of the limitations of the entropy is that it can't show how much the classification accuracy is reasonable [8] . Since the mixed pixels have high entropy it can not demonstrates classification accuracy. Therefore, in the presence of mixed pixels the high entropy always indicates the best classification accuracy. Other limitation of entropy is that it can not show how much the classification accuracy is best or poor. Therefore, entropy parameter can not be used to compare the accuracy of two classification procedures. Therefore, Foody [5] propose that we can use of crossentropy for sub-pixel classification accuracy, if a subpixel or fuzzy ground truth map exist. Cross-entropy parameter is determined as the following equation: p(x p ) = The posterior probability of the classification result in pixel x p p'(x p ) = The posterior probability of the pixel x p in ground truth map Cross-entropy is calculated from the probability distributions of class membership derived from the remotely sensed and ground data sets. The use of crossentropy as an indicator of classification accuracy was investigated with reference to land cover classifications of two contrasting test sites. Thus cross-entropy is calculated for each pixel and defined as the expected information content of a piece of information that would reveal its true class. The major problem of this method is that it needs the fuzzy ground truth map; the matter which often is hard to be available.
One of the accuracy assessment methods is comparative evaluation between area of ground truth and area estimation via classifiers [1] . In this method the area of each class is obtained by the appropriate fraction map. These values are compared with the same other areas which obtain from reliable sources (e.g., Old maps, database or other classifications) [3] ; More similar values the more accurate classifications. This approach uses the fraction maps to calculate the area covered by each class. In this approach we just sum the fractions of each class ignoring the spatial distribution of the errors. The nonsite-specific nature of this approach is, however, a major limitation as a map could easily display the classes in the correct proportions but in the incorrect locations [6] .
Additionally result of this method give any accuracy parameter that can be used at the comparing two or more classifications. Logically the closeness of the estimated and true area is the basic criterion for accuracy of the classification. Thus relatively we can just say "this classification is more accurate than the other one".

Correctness coefficient as a new accuracy method:
As mentioned previously, we can not use traditional accuracy assessment procedures for sub-pixel accuracy assessment. For accuracy assessment of this kind of classification results we have to use fraction maps as the main results of the sub-pixel classification.
In the first steps a parameter which expresses the matching rate of the results (fraction maps) with the ground truth data is needed. For this purpose, we introduce the Correctness Coefficient (CC) parameter. In order to obtain correctness Coefficient, a binary map for each class is generated from the ground truth data. The binary maps of classes are determined as the following expression: In fact by this multiplication, for each pixel with value 1, the calculated fraction remains and zero components of the binary map dismiss the other fractions which have no any corresponding ground truth data. So, in this approach for each ground truth pixel of a particular class, the relevant fraction value will be remained. As a result, it make possible to calculate the correspondence of the resulted fraction with the ground truth data. After this step, correctness coefficient of each class, S i , is estimated as follows: where, Ng is the number of known pixels in the ground truth map. CC also can be computed for each class individually: Where NP i is the number of known pixels for ith class in the ground truth map. Correctness coefficient is expressed as the percentage and resembles the overall accuracy in the traditional error matrices and it can be used as an overall accuracy measure for the sub-pixel results.
In addition to the accuracy parameters, some error measures can also be derived to express the contained errors in the results. Since the commission and omission error are defined in the traditional accuracy assessment, we introduce these parameters on the basis of the ground truth binary maps and classification resulted fraction maps. In a traditional error matrix, commission errors define the percentage of those pixels that have been labelled as a particular class but in ground truth are in a different category. Analogously omission defines the percentage of pixels from a particular class which have been labelled as the other classes. By this concept we can calculate omission and commission errors for each class using sub-pixel classification results. Firstly we subtract each fraction map from the binary map for each class individually:

Evaluation and experiments:
In order to evaluate the CC method, a sub-image of the Airborne Visible/Infrared Imaging Spectrometer (AVIRIS) of hyper spectral data was taken over an agricultural area of California, USA in 1994. This data has 220 spectral bands about 10 nm apart in the spectral region from 0.4-2.45 µm with a spatial resolution of 20 m. The image has a size of 145 rows by 145 columns. Figure 3 and 4 show the colour composite of the study area and its ground truth map respectively.
The study area consists of 16 classes. Since, there is only12 classes in the reality regarding the radiometric overlap between classes some of the (spectrally) similar classes are merged. As a result, 12 distinct spectral classes remain for experiments. This study can not affect to the obtained results. Table 1 shows the number of pixels for each class in the ground truth map. In addition, ENVI and Matlab software's were used for implementing and evaluating the algorithm.
The basic assumption of the Linear Spectral Unmixing (LSU) is that the most of pixels are mixtures of objects. Once all the end members are found in an image (using the mean Regions Of Interest (ROIs) procedure), all the remaining pixels are considered to be linear combinations of these endmember pixels.   Some of the hyperspectral image bands are the subject of absorption and they contain only little signal and more noise. Analysis of the original spectral bands not only is inefficient but also tends to create poor results [3] . The correlation and noisy bands are removed from the original bands by performing Minimum Noise Fraction (MNF) transformation. Moreover, the bands are sorted according to their variances. The output results show that almost 99% of total variance is presented in the 25 first components. The cut off band number was determined by checking the MNF eigenvalues plot that is shown in Fig. 5. Having MNF transform, the data space can be divided into two parts: one part associated with large eigenvalues and corresponding eigenimages and the other part having near-unity eigenvalues are noise dominated images. Visual assessments of images show that the bands numbered greater than 150 consist of data which dominated by noise. Figure 6a shows the first three bands (1, 2 and 3) of MNF images which have large eigenvalues and corresponding eigenimages. Also, Fig. 6b shows the last three bands (193, 194 and 195) of MNF images which are dominated by noise.
In the bands with a large variance, the features can be distinguished from each other in a better mode, therefore classification accuracy increased sensibly. Also, performing MNF transformation on the hyperspectral images reduces the dimensionality of hyperspectral image data for the next processing steps. Hence, this method was applied upon MNF images which lead to the better results than the case which uses the original bands [3] . Table 2 shows that the RMS error of each pixel in the model is decreased by increasing the number of bands.  ------------------------------------- This decreasing of RMS error is obviously observed in the MNF images. The linear unmixing method assumes that the most of pixels are mixed. In the other hand, once all the endmembers are found in an image, all the remaining pixels are considered to be linear combinations of these endmembers pixels. In this procedure, 20, 50, 87, 120, 155, 175 and 195 bands were contributed for classification. Figure7 shows some fraction images that obtained by LSU. LSU classification does not give label directly to any pixel. However, a classified map, which is a pixelbased representation of classification, can be generated by performing maximum value role on the fraction images. In the other hand, by conducting this rule on the fraction maps, a thematic map is generated which each pixel has a specific class label (harden result). This map can be used to compare the proposed sub-pixel and traditional pixel based accuracy assessment methods. Hardening the LSU results can lead to some lose of the information and accuracy but this is one of the common approaches to evaluate the accuracy of the sub-pixel classification results and also we used it only for the comparison of the two methods.
Therefore, the mentioned accuracy measures for the resulted fraction maps are computed using ground truth data (Fig. 4). The classification accuracy increased by increasing the number of bands which are contributed in the classification algorithm. A binary map from the ground truth data is generated to obtain CC parameter for each class. The numbers of binary images are equal to the number of classes or end members. In this research, 12 binary images are produced. The values of binary image are 0 or 1 where number 1 indicates that pixels belongs to the corresponding class of binary image and number 0 indicates that pixels belong to other classes.
AMi is achieved by Pixel to pixel multiplying of ground truth binary image and corresponding fraction map. For CC calculation, the sum of the pixel values in the AMi image is divided to the number of pixels in the land use for each fraction map. For LSU, the achieved results of CC are between 0 and 1.  If CC is equal to 1, it means that all pixels in land use and fraction map have the same label. As a result, the unmixing correctness becomes 100%. In contrast, if CC is equal to zero it means that pixel in land use and fraction map have not the same label. Hence, the unmixing correctness becomes 0%. Intuitively, this parameter expresses the matching rate of unmixing results with the ground truth data. In this research CC and total CC are obtained for each class and LSU algorithms respectively. The results of this operation are shown in the Table 3.
In Table 3, (NP i ) is number of pixels in ground truth and (FM i ) is number of pixels which gathered from the CC parameters on the fraction maps. In the pixel based accuracy assessment methods, Overall accuracy is simply the sum of the correctly classified pixels or samples divided by the total number of pixels or samples in the ground truth. By using of this concept, we can estimate Overall Accuracy (OA) for LSU classification and sub-pixel accuracy method.  By applying of the values of Table 3 is estimated overall accuracy for LSU method. The value of overall accuracy is 89.05%. Furthermore, the confusion matrix is achieved by applying the Maximum Value Rule (MVR) on the fraction maps. Hence, users and product accuracies are computed and their results are shown in Table 4. The OA of LSU is 86% and the corresponding Kappa coefficient is about 84%.
This research attempts to introduce a new method called Correctness Coefficient (CC) for accuracy assessment of the sub-pixel classifiers. Using the concept of matching rate between fraction maps and ground truth data the overall CC and individual CCs have been computed. Also, these parameters are calculated via confusion matrix which is achieved by conducting maximum value rule on the fraction maps. The obtained overall CC over LSU method using 120 bands is about 84.9%. Moreover, the obtained CCs for individual classes and omission errors are shown in Table 5:

DISCUSSION
The experimental results show that CC of some classes such as alfalfa and grass pasture are smaller than other classes. This may due to the spectral similarity between these classes with grass classes that are shown in Table 4. In contrast, compared to mixed classes like soybeans and corn classes some distinct classes such as Hay windowed and oats are classified with high accuracy. The obtained overall CC over LSU method using 120 bands is about 84.9%. In contrast, the obtained results in terms of OA and Kappa coefficient over LSU method which are achieved by MVR on the fraction maps are 86 and 84% respectively. The Kappa coefficient value is close to overall CC of LSU method. It demonstrates that the CC method as an accuracy assessment parameter of a soft classifier can be substitutes reasonably by traditional accuracy parameters. Despite the remarkable results of LSU, the use of soft classifiers is still limited by the lack of wellassessed and adequate methods for evaluating the accuracy of their outputs. CC method uses the fraction maps to extend the applicability of the traditional error matrix method to the evaluation of soft classifiers. In fact it is designed to deal with those situations in which classification and/or reference data are expressed in multimember ship form and intrinsically vague classes.

CONCLUSION
In this study, an innovative accuracy assessment method for sub-pixel classifiers called CC is proposed. Traditional accuracy assessment methods have been designed on the basis of the hard classifiers and therefore there is no straightforward approach for soft classifiers. In the other hand, ground truth data that are inherently hard in nature force the accuracy assessment to be hard.
Recently, few methods have been proposed for the accuracy assessment of sub-pixel classifiers. These approaches can not certify desired flexibility and consistency. Correctness coefficient, recommended by the authors is one of the efforts to ensure the flexibility and consistency of the sub-pixel accuracy assessment regarding the type of the available data and classification methods. Obtained results from the case study show that correctness coefficient (soft) and overall accuracy and kappa (hard) are approximately the same or similar in value enough. The proposed method for the accuracy assessment of the sub-pixel classifiers make possible to inspect the classes individually (individual CCs). Furthermore, each class can be investigated individually in respect of the corresponding commission and omission errors.
Although in this approach we used hard ground truth data, but this is the common case and there is no any fuzzy ground truth in the traditional image analysis. However, using the correctness coefficient parameter and its family we can take into account the major subpixel properties of such a classifiers and the effect of the inherent limitation of the hard ground truth data can be reduced.