Analysis of Artificial Neural Network Point Forecasting Models and Prediction Intervals for Solar Irradiance Estimation

: An accurate knowledge on solar irradiance prediction is particularly required for proper development and planning of Photovoltaic (PV) energy systems. The main purpose of the present research is to assess the accuracy of Artificial Neural Networks (ANN) short-term forecast of univariate solar irradiance time series, with conventional point prediction and Prediction Intervals (PIs), comparing models. The Lower Upper Bound Estimation trained with Particle Swarm Optimization (PSO-LUBE) was used for PIs estimation. Solar irradiance data collected from a station in Amazon region in Brazil was used to train and test the models. Results demonstrate that all ANN models yield good accuracy in terms of prediction error: 8.1-8.5% for normalized root Mean Square Error (nRMSE), 5.8-6.0% for normalized Mean Absolute Error (nMAE) and 94-95% for determination coefficient (R 2 ). However, due to the accuracy of PI information (Coverage Probability = 94.94% and PI Normalized Average Width = 32.50%), PSO-LUBE was the best method tested for decision-making.


Introduction
The energy crisis has become a major concern for all countries around the world due to increasing energy demand. Consequently, Renewable Energy Sources (RESs) have become significantly important in the 21st century due to environmental pollution and depletion of fossil fuels (Qazi et al., 2015). Solar energy is one of the promising alternatives, particularly in Brazil since most of its territory is at the tropical region (Ferreira et al., 2018). Solar power also plays an important role for electrification in remote communities in the Brazilian Amazon region, which are not reached by the Brazilian interconnected electricity distribution system (Lima et al., 2016).
The power produced by Photovoltaic (PV) generators depends mainly on the absorbed solar irradiance. However, solar irradiance on a panel varies with geographic location, time, orientation of the panel and shading conditions. Such factors explain the variable and intermittent behavior of solar power. Therefore, the prediction of solar irradiance is advantageous for efficient PV power generation.
Accurate forecasts of solar irradiance data are important inputs for successful planning and decisionmaking processes. Thus, in recent years several works focus on improving the quality of forecast methods and developing new ones, aiming at more reliability in the estimation of outcomes. Particularly Artificial Neural Networks (ANNs; Haykin, 1998) have been applied to several problems related to PV solar energy, for example maximum power point tracking (Mitsuya and Meneses, 2019) as well as solar radiation forecasting (Khosravi et al., 2018).
When ANNs are used for point predictions no information about the uncertainty in the data is given. Although point forecasting is successfully used for different applications, complementing results with measures of uncertainty is particularly interesting.
In order to cope with that issue, Prediction Intervals (PIs) may be applied to forecasting problems. PIs provide an adequate quantification of uncertainty, in 348 which the desired value in the forecast will be within the limits of such intervals, with an associated confidence level (Khosravi et al., 2011a).
Due to the importance of forecasting the solar irradiance and the need of representing the uncertainty related to the prediction, our work aims to evaluate and compare different point forecasting accuracy in relation to PIs estimation. For that purpose, three ANN models, namely Multilayer Perceptron (MLP), Elman (ELMAN) and Non-linear Auto-Regressive (NAR) networks were implemented for point predictions, as well as the Lower Upper Bound Estimation (Khosravi et al., 2011a) with an ANN trained with Particle Swarm Optimization (PSO-LUBE) was implemented for determining PIs for a univariate solar irradiance time series, with data from the Brazilian city of Belém, in the state of Pará, located in the Amazon region.
The main contributions of the present work are: (i) Assessment and comparison of the accuracy of the estimated PIs with the point forecast provided by conventional ANN models and (ii) forecasting of a univariate solar irradiance time series for a real case study, located in the Amazon region, with a great solar energy potential and variability due to clouds, providing information for future studies.

Artificial Neural Network
An ANN is an information processing system inspired by the human brain architecture (Haykin, 1998), able to learn complex and non-linear relations among variables from observed data. The ANN is defined as a distributed system with simple processing units, called neurons. During training process, those neurons have the natural propensity for storing experiential knowledge and making it available (Haykin, 1998). ANNs are based on training algorithms that adjust the synaptic weight values to approximate to an expected output, when certain targets are given. Several works discuss the successful application of Artificial Neural Networks (ANNs) in solar irradiance forecasting (Voyant et al., 2017). Kashyap et al. (2015) developed and compared ANN models to predict solar radiation in India. They varied the architecture parameters for eight ANN models for comparison, with 10 meteorological parameters in subcategories. In such work, the authors obtained a Root Mean Square Error (RMSE) around 25-30% for the best model with an Elman network. Wang et al. (2016) compared three types of ANN models considering meteorological variables (air temperature, relative humidity and other estimates) from different stations with different climate zones as input to the models. The results indicate that the Multilayer Perceptron (MLP) provides the best accuracy in such case.
Two MLP models with four different training algorithms were investigated by (Premalatha and Armirtham, 2016) for the prediction of monthly average global solar radiation. Models were trained with data respectively from four and five different locations in India. Such data were collected during 10 years. A good agreement was found between predictions and real values, especially in the case of the second ANN model, which used data from all locations. Renno et al. (2016) developed two models based on ANN for irradiance for a solar application to a residential building. In order to evaluate the performance of the models the MAPE, RMSE and R² were proposed as metrics. The best ANN configuration to predict the solar irradiance was obtained with MAPE = 4.57%, RMSE = 160.3 Wh/m 2 and R 2 = 0.9918. Alsina et al. (2016), developed an ANN model for prediction of monthly average daily global solar radiation over Italy, considering 17 locations for training and 28 locations for testing. The best ANN used 7 input parameters, with resulting MAPEs ranging from 1.67 to 4.25%.
Regarding the construction of PIs by ANNs trained with PSO, (Quan et al., 2014a) applied PSO-LUBE to six datasets, improving the results obtained by (Khosravi et al., 2011a;2011b), with a much faster computation. Quan et al. (2014b) obtained high quality PIs in the prediction of short-term load and wind power with PSO-LUBE.

Prediction Interval (PI)
Several time series are predicted with no reference with respect to accuracy (Chatfield, 2000). Interval forecasts such as the Prediction Intervals (PIs) are interesting for coping with such issue, providing an assessment of future uncertainty in different scenarios.
PIs consist of lower and upper bounds that a future unknown value will lie within, with a predetermined probability, called confidence level. Different from point forecasting, PIs are more reliable and informative for decision-makers, giving support to select the best action under uncertain conditions.
The LUBE method combines CP and PIW in one single quality measure of optimization. The cost function developed (Khosravi et al., 2011b) examines at the same time coverage probability and width and it has been successfully applied to different forecasting situations. However, the cost function is nonlinear, complex, discontinuous and non-differentiable. In the literature, optimization methods are applied to minimize this cost function, such as Genetic 349 Algorithms (Ak et al., 2015;Khosravi et al., 2011a;2011b) and Particle Swarm Optimization (PSO) (Quan et al., 2014a;2014b;Galván et al., 2017;Sousa et al., 2017). PSO has strong search ability for optimization and in the present work, PSO was used in the optimization of the LUBE method's ANN.
Several ANN-based methods for the construction and assessment of PIs have been proposed in the literature. Khosravi et al. (2011b) proposed a simple and fast method, called Lower Upper Bound Estimation (LUBE) that applies a fully connected feed forward ANN with two outputs to directly generate the bounds of PIs. Assessment of the quality of constructed PIs is made with two quantitative evaluation indices: Coverage Probability (CP) and Prediction Interval Width (PIW). Higher CPs and narrower PIWs are expected for high quality PIs.
A PI is described by an interval that include an unknown target value with a certain probability, called confidence level (1-α) (Ak et al., 2015) and it is expressed as: where, L(x) and U(x) are the lower and upper bounds, respectively, of the estimated PI of the output y(x) corresponding to the input x. The confidence level, generally equal to or greater than 0.9, is the reference for the expected probability that the next measured value or the series lies between the boundaries of the PI. CP and Prediction Interval Width (PIW) are the metrics that evaluate the quality of constructed PIs. CP indicates the probability that the target values will lie within the upper and lower bounds and it is considered as the calibration of the PI quality, given by: where, np is the total number of samples and ci is a Boolean variable, which shows the coverage behavior of PIs. If yi[L(x), U(x)], ci = 1, otherwise ci = 0. In practice, to consider the PIs as valid, the CP should be equal or greater than the nominal confidence level.
If the width of intervals is too large, they convey little information about the targets, which is not useful for decision making. Thus, the Prediction Interval Normalized Average Width (PINAW) has been used to evaluate the width of PIs and it measures the average width of PIs, for all points in the dataset, as a percentage of the underlying target range (Quan et al., 2014a).
where, Li and Ui are respectively the PIs' lower and upper bounds and R is the range of the underlying targets.
In point forecasting Mean Square Errors (MSE) are commonly used for evaluating training models. Thus (Quan et al., 2014b) proposed a new width assessment index used for training ANNs models. The PI Normalized Root-Mean-Square Width (PINRW) is described by: where, Li, Ui and R are the same as in PINAW. In the present work, PINRW is used for training the ANN model while PINAW is used for testing. The cost function proposed by (Khosravi et al., 2011b) is called Coverage Width-based Criterion (CWC), which presents a comprehensive balance between CP and PINRW (in the case of testing, PINAW instead of PINRW). It is defined as: where,  and  are the nominal confidence level and the scaling factor that magnifies the difference between CP and , respectively. The function  (CP) is equal to one during training, while in testing it is a step function whose value depends on CP, that is,  (CP) = 0 when measurement of CP is not less than  and  (CP) = 1 otherwise, penalizing the function CWC.

Particle Swarm Optimization
Particle Swarm Optimization (PSO; Eberhart et al., 2001) is an algorithm based on social and cooperative characteristics of several species in the nature. The swarm particles are guided by personal particle experience and global swarm experience, which act as attractors towards optimal or near-optimal positions. In the PSO, the position of a group of particles (swarm) is initialized randomly. Each one of the positions represents a candidate solution to the optimization problem. Each particle has its velocity and position value updated and the best positions of each particle best P and best global solution best G act as attractors for the other particles. The canonical formulas for velocity and position update are defined as: and:

350
where, k represents the k th iteration,  is the inertial weight, c1 e c2 are acceleration coefficients, r1 and r2 are uniformly distributed random numbers, , best i P represents the best individual position of the particle i and best G represents the best position of the entire swarm. In this study, the canonical PSO is integrated into the LUBE method, in order to construct PIs.

Methodology
In this section we present the case study and corresponding dataset, the ANN models used and the performance evaluation criteria. Regarding the analysis step, basically we present the results in two parts. In the first part we compare the point forecast ANN models (MLP, ELMAN and NAR) with the PI Mean Values (PIMVs), that is, for each PI predicted by the LUBE method, the mean value is considered for calculating the metrics nRMSE, NMAE and R 2 , as if the PIMV were a point forecasting result. In the second part, percentage intervals are attributed to the point forecast results, in order to compare them to the PIs generated by the LUBE method.

Case Study and Dataset
The solar irradiance time series data used in the present work was obtained in the city of Belém (1°27'S, 48°29'W), the capital of Pará state in Brazil which is located in the eastern Amazon region. The data was collected by the Group of Studies and Development of Energy Alternatives of the Federal University of Pará (GEDAE-UFPA) from December 2015 to November 2016 (366 days), measured by a pyranometer (CM 11 Kipp and Zonen) at the site every 10 minutes. Regarding the data preprocessing, data belonging to the time interval from 5 am to 7 pm were considered, thus excluding night hours, with absence of solar irradiance (Voyant et al., 2014).
Temperature, relative humidity, air pressure and wind speed were also collected for the same area in the same period. In order to perform an exploratory data analysis, the Pearson correlation coefficient was calculated. The correlation coefficient matrix obtained is shown in Table 1. The irradiance has moderate positive correlation with temperature variable. It is also possible to see a negative correlation between the variable humidity with the irradiance.
In the present work the sliding window method was used (Paoli et al., 2010), for both point and interval prediction.

Artificial Neural Networks
In the present work three ANN models were applied to univariate forecast of solar irradiance, namely MLP, ELMAN and NAR. Different configurations were preliminarily investigated for the ANNs in order to find the best performances. The input data was divided sequentially as 80% for training and 20% for testing. After splitting, these two sets were normalized between 0.1 and 0.9. According to (Ak et al. 2015), normalization leads to better results.
The main characteristics chosen for those ANNs are one hidden layer, the activation functions are the logistic function to the hidden layer and linear function in its output layer, both transfer functions are chosen because better results were achieved with them in our preliminary tests and trained with LM learning algorithm. This algorithm is suitable for this type of study of time series prediction (Neelamegam and Armirtham, 2016; Voyant et al., 2017). The training gain, μ, is a parameter that measures the adapting and learning rate of the model, in the range of [10 3 , 10 10 ], μ decrease and increase ratios are 0.1 and 10, respectively. Table 2 shows the training parameters obtained in the preliminary tests.

351
In order to assess the robustness of the results and the reproducibility of the algorithms, each model simulation, with the same parameters, was run 10 times with weight values were randomly initialized.
In the present work, all algorithms have been implemented in MATLAB®, using its Neural Network and Optimization toolboxes. The ANN models are described in the following subsections.

Point Forecast ANN Models
In order to predict the solar irradiance, a fixed number of previous solar irradiance observed values was considered as input for the models, in each training process (using the sliding window method). The number of previous observed values in the window was determined by the analysis of the empirical Autocorrelation Function (ACF; Chatfield, 2000). The ACF measures the correlation between data of a time series, that is, how much the current data is correlated to next time steps. The ACF for the irradiance time series data is shown in Fig. 1. Figure 1 shows a non-negligible correlation of the solar irradiance time series that is an indication of the adequate number of previous observed values to be used in the input layer. For the ELMAN network, initially only one neuron was considered in the input layer, in which it would provide information to predict the next irradiance value, but after the results presented overfitting, we added one more input, which resulted in improvement in the training and validation phases. Finally, for NAR network, only one input was fixed to forecast the irradiance.
The number of hidden neurons and delays was defined in a trial-and-error basis. The MLP ANN had one hidden layer and the number of neurons was varied from 2 to 20. The ELMAN network structure was tested, varying the number of delays from 2 to 10 and the number of neurons in the same range, in one only hidden layer. NAR network was defined with just one hidden layer and neurons and delays in this layer were varied from 2 to 20. Table 3 presents the number of neurons in the input layer (Ni), number of neurons in the hidden layer (Nh) and number of delays obtained in the preliminary tests. For MLP network, the best structure presented eight neurons in the hidden layer and four inputs of the irradiance time series. The ELMAN network was achieved with a configuration of five hidden neurons and eight delays. The best result with a NAR network was found with a 1-3-1 architecture and nineteen delays. As mentioned before, the LUBE method uses a feed forward ANN model with two outputs to directly construct PIs in one step. After a set of different architectures, the best PIs results were obtained with a 3-11-2 configuration. Then, from the PIs calculated by PSO-LUBE, the PIMVs were calculated for a comparison to point predictions ANNs.

Interval Prediction: PSO-LUBE Method
The Lower Upper Bound Estimation (LUBE) proposed by (Khosravi et al., 2011b) is a method that adopts an ANN with two outputs to construct PIs. Figure  2 illustrates how an ANN is used in LUBE method, with two output neurons (one for the lower bound and other for the upper bound of the PI). Figure 3 shows the flowchart of the PSO-LUBE method.
For PSO-LUBE implementation details, the interested reader is referred to (Quan et al., 2014b). The parameters of PSO c1, c2, wmax and wmin were configured with the values 1.22, 1.49, 0.9 and 0.7, respectively. In the cost function CWC α = 0.1, µ = 0.9 and  = 50. The number of particles was 50 and the maximum iteration was 2000. In the present work, the expected output is the solar irradiance prediction interval, which provides information about the uncertainty associated with the forecasts, also enabling scenarios for the best and worst irradiance conditions.

Performance Evaluation Criteria
In order to correctly evaluate the performance of the ANN point forecasting models and the related errors, three different metrics were considered, namely normalized Root Mean Squared Error (nRMSE), Normalized Mean Absolute Error (NMAE) and determination coefficient (R²) are considered (Khosravi et al., 2018;Leva et al., 2017). In addition, these metrics are also computed for the PIMVs, defined by the difference between upper and lower bounds of the estimated PIs and then PIMVs were compared with the accuracy of the ANN prediction models. These metrics are calculated by:

353
where, yi, ˆi y , y and n are respectively the measured value, the predicted value, the mean value and the total number of samples.
The Kruskal-Wallis nonparametric test is used to compare different independent samples, indicating if there is a statistically significant difference between at least two of them. In our case, the Kruskal-Wallis test was used to assess significant differences in the predictions of solar irradiance obtained between the models, considering the nRMSE results of each network. The Dunn's test was the post-hoc test used for pairwise comparison (Dmitrienko et al., 2007). A significance level α = 0.05 was used as a threshold for the statistical tests. Table 4 shows the results of the metrics nRMSE, NMAE and R² averaged over 10 experiments. The ANN with the lowest average nRMSE and highest R² was ELMAN network (respectively 8.184% and 0.949). The lowest average NMAE was obtained with NAR (5.873%). PSO-LUBE PIMV produces metrics results similar to the other models, although slightly different. It is important to remember that PIMV forecast corresponds to one scenario that can be obtained from the interval forecasting performed using the PSO-based LUBE method. Likewise, the upper and lower bound can also be used as best and worst cases scenarios for solar irradiation and this is particularly important for decision-making. Despite relatively approximate results, Kruskal-Wallis and Dunn's statistical tests were still necessary for further analysis, as we will discuss later in this subsection. Figure 4 shows a comparison of the time series test data observed for solar irradiance forecasts provided by ANNs and the PIMVs obtained by PSO-LUBE. Figure 4 also shows that all predictions approximately follow the real data (black line). Those predictions practically overlap each other in most of the time interval presented. Figure 5 shows the nRMSE box-plots for MLP, ELMAN, NAR and PSO-LUBE's PIMVs, for the testing phase (that is, for data which the algorithms were not trained with) for 10 tests of solar irradiance prediction by each method.

Part 1: Point Forecasting Results
The point forecasting methods and the PSO-LUBE PIMVs had very similar performance for the irradiance time series prediction, thus statistical tests were applied to nRMSE in order to assess whether there was statistically significant difference in such results. Then the Kruskal-Wallis and Dunn's tests have been used. Kruskal-Wallis test presented a statistically significant difference between the models' results (p < 0.0001;  2 = 35.27) and the null hypothesis was rejected. The Dunn's test was applied showing that there is only statistically significant difference between ELMAN and PSO-LUBE. The other pairwise comparisons failed to reject the null hypothesis. In other words, PSO-LUBE PIMVs was only outperformed by ELMAN network.
Considering only the ANN point forecasting models, the results are consistent with the related literature, although there is no statistically significant difference between MLP, ELMAN and NAR.

Fig. 5: Boxplot of the nRMSE metric
Although PSO-LUBE PIMV is outperformed by ELMAN, its advantage in obtaining predictions with information related to uncertainty is still remarkable and will be discussed in the next subsection.

Part 2: Interval Forecasting Results
The training process of the LUBE method by the PSO algorithm presented a good convergence, obtaining in the test phase a high value for CP and lowest possible value for PINAW and CWC, as we can see in Table 5. For the real data, the PIs constructed by PSO-LUBE cover the targets in a great percentage, which implies that the CP index reaches close or more than the confidence level (90%). Figure 6 shows the constructed PIs estimated by the PSO-LUBE model. We can observe that the target samples (black line) lie within the constructed lower and upper (brown and green dashed lines) bound in almost all data seen in the plot, as reported by a CP index of 94.94%. In other words, approximately 95% of the test data lie between the limits of the estimated PIs, confirming that the prediction given by the PIs ensures an optimal confidence level about the future values and the width of PIs is determined by the uncertainty level present of the dataset, related in this case of solar irradiance, especially for the geographical area of interest, by intermittent changes according to weather conditions in different time scales. Figure 6 shows the point predictions of MLP, ELMAN e NAR models as well as the PSO-LUBE PIs. The PSO-LUBE CP index was calculated considering not only the real data, but also each of these forecasting outcomes, in order to evaluate the coverage of the PIs constructed in relation to the other predictions for the case study. Therefore, these ANN point predictions are encompassed by the upper and lower bounds of the PI, resulting in CPs of 95.95%, 90.14% and 94.50% for MLP, ELMAN and NAR, respectively. All the CPs are higher than the pre-established confidence level for this problem. Therefore, all these ANNs point predictions would be considered as scenarios that could be predicted directly in a single forecasting in PIs form, with high probabilities of success.
Finally, percentage intervals were attributed from the ANNs predictions (±10%, ±30%, ±50% and ±70%). Then CP and the average width of the predictions were calculated. Table 6 shows the results for those percentage intervals.
Firstly, it is important to emphasize that a narrower interval forecast is not necessarily better, mainly if the confidence level is not been reached by the CP index. According to Table 6, all the intervals created with an increment and decrement of 10% presenting very narrow widths, but just a few target data is inside their bounds, with CPs corresponding to 37.87%, 37.51% and 36.84%. Therefore, this interval of forecasts cannot be considered as valid since it is very below of the confidence level. It was noticed that the larger the effect of uncertainty in the point forecasting, the wider were the intervals generated by the percentage variation. Looking at the 50% intervals, although they have obtained width values close to the PINAW (32.50%), of 36.10%, 35.95% and 36.09% to MLP, ELMAN and NAR respectively and they still presented a CP below the pre-determined confidence level.
Regarding the 50% percentage interval approach, Fig.  7  355 that the CP of these intervals are far worse than the original PIs. Regarding the intervals created with a 70% variation, low CPs are also obtained as well as very large average widths (approximately 50%) compared to 32.50%, which is the LUBE PINAW (Table 5). Therefore, PI forecasting provides reliable information and with a high accuracy for the dataset tested. In other words, PI can yield high precision point forecasting using PIMVs, although from a point prediction it is not possible to yield accurate intervals of prediction. Therefore PSO-LUBE and the uncertainty information associated are helpful and valuable in decision-making.

Conclusion
In the present work, ANN models and a PI construction method were used for forecasting a univariate solar irradiance time series. Data was acquired in the Brazilian city of Belém, in the state of Pará, located in the Amazon region.
MLP, ELMAN and NAR neural networks were used to obtain point predictions and the PSO-LUBE method was used to obtain PIs.
In the first part of the results (point forecasting analysis), we found statistically significant difference between ELMAN point prediction and the PIMV estimated by the PSO-LUBE method. There was no statistically significant difference between PSO-LUBE PIMVs, MLP and NAR networks.
In the second part (PIs analysis), concerning the PIs estimated by the PSO-LUBE method in relation to the real data, CP = 94.94% and PINAW = 32.50% were achieved. CP was above the pre-determined confidence level. PSO-LUBE also covers 95.95%, 90.14% and 94.50% respectively of MLP, ELMAN and NAR point predictions. Percentage intervals generated from the ANN point predictions have poor performance.
In summary, PSO-LUBE is statistically as good as MLP and NAR point prediction when its MVPI is compared with the ANNs' results. In relation to the PI CP values regarding real data and point predictions, PSO-LUBE found values greater than the 90% confidence level. When intervals are generated from point predictions, PSO-LUBE obtains the best CP.
In conclusion for situations that require analysis of forecasting scenarios, estimating PIs with the LUBE method has become the best alternative to predict solar irradiance for providing information for decisionmakers, allowing different strategies to be planned for the range of possible outcomes indicated by the interval prediction.