Combining Model-Based Testing and Failure Modes and Effects Analysis for Test Case Prioritization: A Software Testing Approach

: Due to The complexity of modern software projects and the increasing size of software systems, it becomes difficult to manually perform tests with limited resources. Also, manual testing cannot assure that the software is tested using all possible combinations of inputs. Therefore, automate software testing activities have become primordial in the Software Development Life Cycle (SDLC). Model-based testing is a prominent validation technique in software testing that uses models of the system under test to automatically generate test cases; this generation leads to a large number of test cases; which cannot exhaustively be executed, due to time and cost constraints. Moreover, the test-cases execution order has an influence on the rate at which faults can be detected. Therefore, it is essential to prioritize test cases in order to execute the most important with limited time and cost. On the other hand, The Failure Mode and Effects Analysis (FMEA) considered as one of the formal techniques to do risk based testing. It offers a structured methodology to identifying the system failure modes, analyzing their effects and setting up controls to risk reducing and improve the quality of systems or products. By applying such a technique to Model-based testing projects, we can benefit from FMEA analysis and the automating in same project. Through this paper, we introduced a new model based testing approach for prioritizing and ranking test cases according to the requirements and failure modes. In the suggested approach we used the Failure Mode and Effect Analysis methodology in model based testing way to automatically generate a set of pair {test case, priority number} to prioritize test cases. It differs from existing approaches in that it generates two types of test cases; requirement-based test cases and failure mode-based test cases. For the purpose of experimental evaluation and analysis, we will compare the suggested approach with some well-known prioritization methods and we will present a case study to illustrate the potential application of the proposed approach in a future work.


Introduction
Software testing is a process of verifying and validating that a software system works as expected and meets the technical and business requirements. It is considered as an important activity to assess software quality. This activity has always been a time and cost consuming task. In this context, much research has been performed aiming at reducing the time and costs of software testing. Thus, model based testing is being profoundly practiced in most of the software industries to achieve this goal. It represents an attractive solution to do software testing.
Model based testing is a software testing technique that relies on models of a system under test and/or its environment to automatically generate test cases for the system (Utting et al., 2012). This generation leads to a large number of test cases; which cannot exhaustively be executed and the execution order has an influence on the rate at which faults can be detected Rothermel et al., 1999). Therefore, it is essential to prioritize test cases in order to execute the most important with limited time and cost. This prioritization of test-cases can increase the rate at which critical faults are detected in the first run.
The main objectives of test case prioritization are to rank the test cases according to an adequacy criterion Elbaum et al., 2001) and to reveal faults earlier so as to reduce the total cost of testing. Rothermel et al. (2001) have defined the test case prioritization problem and described several issues relevant to its solution. This problem is defined as follows. Given: • T: A test suite • T' and T'': Are different variations of the test suite • PT: The set of permutations of T • f: A function from PT to the real numbers, which represents the preference of the tester while testing Problem Find T'∈PT such that: Different test case prioritization techniques are proposed. These techniques can be categorized into seven categories, we find; code-based test case prioritization, model-based test case prioritization, requirement-based test case prioritization, risk-based test case prioritization and model and risk-based test case prioritization, history-based test case prioritization.
This classification is described and detailed in the form of table as show in Table 1.
For CBTCP, the source code of the system under test is used to prioritize the test cases. This category of techniques involve ranking test cases using the number of covered statements, the number of covered lines of code and the number of functions that were exercised by the test case. Table 2 illustrates some existing test case prioritization techniques that are based on the code coverage.    Gökçe et al. (2015), Belli and Gökçe (2010),  Kavitha and Sureshkumar (2010), Ashraf et al. (2012) For MBTCP, a model that represents system behavior is used to prioritize test cases. The model based test case prioritization may improve the early fault detection as compared to the code-based test case prioritization (Korel et al., 2008). Table 3 illustrates some existing model based test case prioritization techniques.
For ReBTCP, requirements information derived during requirement extraction are used to improve the effectiveness of test case prioritization. Table 4 illustrates some existing Requirement based test case prioritization techniques.
For RBTCP, the risk factor is used to improve the effectiveness of test case prioritization. Table 5 illustrates some existing risk based test case prioritization techniques.
For ReRBTCP, the risks associated with software requirements are used for test case prioritization (James, 1999). Table 6 illustrates some existing requirement and risk based test case prioritization techniques.
For MRBTCP, the risk factor and the models are used in same times to improve the effectiveness of test case prioritization. Table 7 illustrates some existing model and risk based test case prioritization techniques.       Table 9 illustrates some existing search based test case prioritization techniques.
In this paper, we proposed a new model based test case prioritization approach. We exposed how to perform test-case prioritization in model based testing way using failure mode and effect analysis technique. The proposed approach is both model-based and failure mode-based. It combines model based testing and risk based testing to automatically generate test cases. The main purpose of this approach is to generate test cases associated with their prioritization to rank test execution, consequently, higher priority test cases are executed before lower priority test cases.
The rest of this paper is organized as follows. Section 2 presents discussion of related works. Section 3, presents the necessary background about model based testing technique and their process. Section 4, describes Failure Mode and Effect Analysis (FMEA) method and its process. In Section 5, the main contribution of the paper is introduced, it presents the proposed approach to the test case prioritization problem in model based testing way. Section 6 discusses the benefits and limitations of the new approach versus existing approaches. Finally, Section 7 concludes the paper and describes future work.

Related Works
In the literature, several techniques for Test Case Prioritization have been proposed. However, , have examined and investigated test cases prioritization techniques for abstract test cases. They have categorized the existing techniques into four categories; a Non-Information-Guided Prioritization category (NIGP), Interaction Coverage Based Prioritization category (ICBP), Input-model Mutation Based Prioritization category (IMBP) and similarity based prioritization category (SBP). The main finding of this work is that ICBP category has better testing effectiveness than other categories. Sultan et al. (2017), have performed an analytical review and comparison of different test cases prioritization techniques. The comparison study have exposed and compared Fault Severity technique carried out by Varun Kumar and Kumar (2010), fault Localization technique proposed by Kavitha and Sureshkumar (2010), Mutation faults technique effectuate by Malhotra and Bharadwaj (2012), Ordered Sequence of Program Elements technique, Average Percentage of Faults Detected (APFD) technique performed by Srivastava (2008), Model Checker technique performed by Korel and Koutsogiannakis (2009;Korel et al., 2008) and Search Algorithm technique proposed by Maia et al. (2010). This comparison is based on key idea, advantages and limits of each technique. Mohanty et al. (2011), have realized a survey of test case prioritization techniques. They have exposed and analyzed some existing techniques in code based, requirement based and model based prioritization. They have exposed methodology and some examples for each category, for example, for code based test case prioritization category, they have presented Srivastava' technique (Srivastava, 2008) which is based on APFD (Average percentage of Faults detected) value, Rothermel' technique (Rothermel et al., 1999), Prashant' technique (Malangave and Kulkarni, 2008) and Li' technique (Li et al., 2007). For model based test case prioritization category, they have presented the techniques performed by Korel and Koutsogiannakis (2009;Korel et al., 2008). For requirement based test case prioritization category, they have presented Srikanth' techniques , Acharya' technique (Acharya and Jena, 2010) and Wu' technique (Wu et al., 2001).
On the other hand, in model Based testing area, Belli and Gökçe (2010), have introduced a new approach that use Event Sequence Graphs (ESG) and focus on model based specification and coverage-oriented testing to rank tests according to their preference degrees. The main advantage of this approach is that no priori information is needed about the tests.
Also, Gökçe et al. (2015), carried out a search for test case prioritization in model-based testing. They have proposed a new approach that is based on eventoriented graph models. In the proposed approach, the prioritization technique is performed by means of cluster analysis. It provides an effective algorithm for ordering a given set of test cases with respect to their degree of preference as perceived by the tester, which results in a set of priority-sorted test cases. The main contribution this study is the introduction of 13 attributes that enable generating test cases from a model hierarchy with several levels as if it is a singlelevel model.
In the same context of model based testing, Panigrahi and Mall (2010), have proposed a new technique to prioritize test cases for regression testing of object-oriented programs. They have proposed a new model named "Extended Object-oriented System Dependence Graph" (EOSDG) to model object oriented programs. This model is based on System Dependence Graph for object-oriented software of Larsen and Harrold. Panigrahi's technique involves constructing EOSDG models for original and modified programs to represent control and data dependences as well as static object relations such as inheritance, aggregation and association. For prioritization of test cases, Panigrahi and Mall have constructed a forward slice of EOSDG to identify directly or indirectly affected model elements and they have constructed a backward slice to identify the model elements indirectly tested by a test case. They have also considered the dependencies among test cases while maintaining the prioritized test suite. This approach results in significant increase in the detection of regression faults arising from object relations as compared to related approaches.

Model Based Testing (MBT)
Model-based testing is a software testing technique that relies on models of the behavior of the system and/or its environment to automatically generate test cases (Utting et al., 2012;Utting and Legeard, 2007;Pretschner, 2005). The model of the System Under Test (SUT) is created mainly by analyzing system requirements. It defines the expected behavior of the SUT with respect to a set of inputs and is provided as an input to a MBT tool to automatically generate a set of test cases. These test cases are executed and compared with respect to the expected results to report any deviation from the expected behavior. In principle, test cases execution activity can also be automated. However, MBT is mainly concerned with the automation of test case generation. This automation decreases the testing time and helps to achieve increased (and measured) coverage of possible execution scenarios. In addition, test models and generated test cases help to document and analyze the system behavior. Changes in requirements can be reflected to models for generating new test cases. Hence, this approach also reduces the maintenance effort. The model-based testing solution consists of producing test cases from System Under Test (SUT) model and/or its environment model by following process that is composed of five main phases (Atifi et al., 2017) (Fig. 1

Failure Mode and Effects Analysis (FMEA)
FMEA has been used in many fields to analyze and manage risk. It was used in the late 1950s to study problems that might arise from malfunctions of military systems, precisely for the flight control systems. It appeared as a formal technique in the aerospace and defense industries. Then it spread to the American automotive industry in the late 1970s. In 1985, FMEA was later adopted by the International Electrochemical Commission. Many authors adapted FMEA to various areas in industries, such as nuclear power industry, environmental concerns, software, semi-conductor, webbased distributed design and healthcare (Ben-Daya, 2009;Ben-Daya and Raouf, 1996). In general, there are many types of FMEA, viz; mechanical FMEA, system FMEA, electrical FMEA, product FMEA, software FMEA, process FMEA, human-use or misuse FMEA and health care FMEA. Although the purposes, terminologies and details of each FMEA type are different, but they share the same basic concept that follows the following process (FMEA General process) (Fig. 2): 1. Identify and choose the items to be analyzed 2. Identify the potential failure mode for each item 3. List the effects of each failure mode 4. Rate how severe each effect and categorize the severity of each failure mode (Assign a severity ranking for each effect) 5. Identify the root causes of each failure mode 6. Assess the probability of occurrence of each failure mode (Assign an occurrence ranking for each failure mode) 7. Identify the controls in place to detect the failure mode 8. Assign a delectability ranking for each failure mode 9. Calculate the risk priority number (RPN) of each effect: The RPN is calculated as the multiplication of the probability index, severity index and delectability index 10. Based on RPN number, identify most critical issues and determine actions needs to be taken into consideration 11. Implement controls and recommendations to eliminate or reduce the high-risk failure modes  In software testing area, FMEA is considered as one of the formal techniques to do risk based testing. It offers a structured methodology to identifying the system failure modes, analyzing their effects and setting up controls to improve the quality of systems or products. FMEA is based on three factors or indexes which are usually evaluated through easily interpreted expressions, each factor is correlated to a score vary between a minimum value and maximum value: • Severity factor (S) that determining the consequence or the cost of the failure mode. It defines the seriousness of consequences of failure effects. Typically, it is rated on a scale of 1 to 5 or 1 to 10, corresponding to negligible or no effects to catastrophic or very high hazardous or effects • Occurrence factor (O) that defines the likelihood of occurrence of a failure mode. It examine cause(s) of each failure mode and how often failure occurs. Typically, it is rated on a scale of 1 to 5 or 1 to 10, with highest occurrence corresponding to highest probability and lowest occurrence to lowest probability • Detectability factor (D) that indicates the probability of detecting the failure. It evaluates the likelihood that a detection method will detect the failure of a potential failure mode before it occurrence. Also, it is ranking in a scale of 1 to 5 or 1 to 10 to rank from lowest to highest detectability The FMEA factors are used to calculate Risk Priority Number (RPN) to measure risk and severity of a failure to prioritize potential failure modes and root causes. To express mathematically, the RPN of an artifact x i can be calculated as follows:

Methodology and General Process
In the proposed MBT approach, the main purpose of performing FMEA is to include risk notion in the MBT concept. It is used to identify potential failure modes for each requirement and to calculate RPN for each failure mode using Severity (S), Occurrence (O) and Detection (D) factors. Potential failure mode is defined as the manner in which the system under test could potentially fail to meet the requirements intent. To offer several choices for evaluating failure modes factors, we chose to follow the usual practice that rates failure modes factors on a scale of one to 10 where 1 is lowest and 10 is highest. Table 10 to 12 list the scales used to measure Severity, Occurrence and detectability factors to calculate RPN.
The risk levels of all identified failure modes can be measured by the three characteristics below: • Frequency of occurrence (probability of occurrence) • The seriousness of their consequences (Severity) • Detectability factor In order to prioritizing the risks we associated with each risk a weight according to five levels (Fig. 3 We can then determine the acceptable combinations (risk/severity/frequency/detectability).  Below average Failure with less moderate impact on system operations (Tolerable impact) 5 Average Failure with moderate impact on system operations (not tolerable in critical situations) 6 Above Average Failure with very moderate impact on system operations 7 High Failure with high impact on system operations 8 Very high Failure with very high impact on system operations (Reduced performance) 9 Extremely High (with warning) Failure with very critical impact on system operations but with warning in advance. 10 Extremely High (without warning) Failure with very critical impact on system operations and without warning. Certain Failure mode very likely to be detected, it will be detected certainly 2 Very high Very high chance to detect the potential and subsequent failure mode 3 High High chance to detect the potential and subsequent failure mode 4 Above average Above average chance to detect the potential and subsequent failure mode 5 Average Moderate chance to detect the potential and subsequent failure mode 6 Below Average Low chance to detect the potential and subsequent failure mode 7 Remote Remote chance to detect the potential and subsequent failure mode 8 Very Remote Very remote chance to detect the potential and subsequent failure mode 9 Almost Uncertainly There is almost no chance to detect the potential and subsequent failure mode 10 Uncertainly There is no chance to detect the potential and subsequent failure mode In our approach we associated a new factor to requirements in order to give a prioritization for each of them. We define it as Requirement Priority Number (RePN).
To calculate the RePN, there are two possibilities according to the generation directives: • If we consider that the requirement with the highest risk is the highest priority requirement, we can calculate the RePN using the risks weights. In this case, RePN represents the average of RPN of failure modes multiplied by their weights associated to a requirement. Mathematically, the RePN for a requirement is calculated as follow: The total number of failure modes for a requirement Or: • If we consider that the requirement that has the average of the associated risks high is the highest priority requirement, we calculate the RePN as the average of RPN of failure modes associated to a requirement. Mathematically, the RePN for a requirement is calculated as follow:

Process Steps
The generic process steps for the new approach are as follows (Fig. 4): Step-1 Requirements management process, which is the first step of model based testing. It consists to collect customer needs, desires and constraints, manage and then classify Category 4: major failure modes, it is necessary to treat them in priority ->> Weight 4" "Category 5: critical and unacceptable failure modes, must be deleted ->> Weight 5" them as requirements. This step is potentially the most important step in any software testing process based on requirements such as MBT. It involves the collection, analysis, prioritization, validation, definition and control of all customer business requirements, it serves to create a requirement repository that is the basis of communication between analysts and testers, is define in a structured way the expected result for the software, in different terms (functional, technical, security, load and response time…). Step-2 Identify potential failure modes for each requirement, i.e. describe ways in which the system might fail to perform its intended requirement. In this step, Items considered could be previous lessons learnt/problems or new issues from brainstorming session. Step-3 Identify the list effects of each failure mode. For each requirement, describe the effects of each failure mode from an internal or external view point. Typically, failure effects could be Safety issues, non-compliance to standards, non-functional features, performance issues, intermittent operations and robustness issues in the system under test. A "failure mode" could have multiple effects.
Step-4 Severity assessment step, which consists to assess the seriousness of the effects of each potential failure mode and then assign a severity ranking for the failure mode. The severity levels can be based on a 1-10 scale as per the guideline in Table 2 (FMEA scale for severity (S)) above. Step-5 Identify the root causes of each failure mode i.e., indicate the weakness that causes the potential failure mode. This step involves constructing a concise, clear and comprehensive list of all root causes of failure mode. Step-6 Assign an occurrence ranking for each failure mode i.e. estimate a likelihood that a specific failure cause will occur (the probability of occurrence). The probability of occurrence can be based on a scale from 1to10 as per guideline in Table 1 (FMEA scale for probability of occurrence (O)) above: Step-7 Identify the controls to detect the failure mode of each requirement i.e., list all existing controls and procedures (inspection and test) that prevent either the cause or the failure Mode. Step-8 Assign a detectability ranking for each failure mode. This step involves assessing the ability of the current control method to detect the failure mode or the failure cause. However we can give a higher value for detectability ranking for a failure mode that can be detected only in testing than the one that can be detected in design itself. Even so, Higher the ability to detect is lower; the value of detection is higher on a scale of 1-10. For example, if the current control mechanism is absolutely certain of detecting the failure mode then the detection would be 1 and so-on as per the guideline in the Table 3 (FMEA scale for Detectability (D)) above. Step-9 Calculate Risk Priority Number (RPN) for each failure mode as product of severity, occurrence and detection rankings. This ranking prioritizes failure modes, namely, the more the RPN number is higher, the risk of that particular failure mode should be treated first. At the same time, problems with low RPN still deserve special attention if the severity ranking is high.
Step-10 Calculate Requirement Priority Number (RePN) for each requirement as average of RPN of failure modes associated to a particular requirement. This ranking prioritizes requirements. Step-11 The purpose of the modeling step in our approach is to model the system requirements and their failure modes. It consists constructing a generic model in testing purpose. This model represents in the same time the requirements of the system under test and their failure modes. It is created by a test analyst using requirements resulting from the requirements management step and potential failure modes resulting from the potential failure modes identification step and annotated by Requirement Priority Number (RePN) and Risk Priority Number (RPN). The test model is described in many ways, depending on the discipline. It can be described by use of diagrams, tables, text, or other kinds of notations. In future works, we will present how to create models that represents requirements and failure modes in testing purpose and how to annotate these models by RePN and RPN. Step-12 The generation step that is realized on the basis of a test generator which takes as input tree elements; the model designed in the modeling step, the test selection criteria selected by the test analyst and the generation directives. The generation directives are new elements in our approach. Each generation directive is classified into one of the following categories follows (Fig. 5): A generation directive is a comment with a special syntax; it can be placed wherever comments are allowed. The main purpose of generation directives is to direct automatic test generation if we want to generate test cases based on the failure modes or to generate test cases based on the requirements or to generate test cases based on both the requirements and the failure modes. The generation step produces the abstract test cases from the test model and a traceability matrix that illustrates the link between test cases and model elements covered by the test cases. The traceability matrix is named as Requirement RPN Traceability Matrix (ReRPNTM), it used to track the relationships between test cases, requirements, potential failure modes, risk priority numbers and Requirement priority number. It links each test case with their requirement, their potential failure modes and their priority number and each failure mode with their risk priority number. The outputs of generation step are; a set of pair {test case, priority number}: The priority number of each test case is inherited from the associated requirement if the generation directive is « requirements » or is inherited from the failure mode if the generation directive is « failure modes ».
In the future works, we will detail the generation directives for the automatic test generation and adopt an existing test generator to take in consideration our inputs to generate appropriate test cases. Step-13 Concretization step, which consists to translate the abstracts test cases to executables test cases in order to be executed on SUT. It consists in making the link between the model elements and the system's concrete elements and involving specific adapters and manual intervention that requires the expertise of the test engineer. Step-14 Execution step, which can be realized manually or automatically by using an automation testing tool. In this phase, the test cases are executed on the system under test by following the prioritization order generated in the generation phase. Eventually the obtained results are then compared with the expected results to give a verdict for each test case and consequently give a status on the operation of the system.

Global Algorithm for FMEA and Model-based Test Case Prioritization
Our proposed GA is shown in Table 13. It is used to schedule the test cases in test suite T based on their potential failure modes, such as the failure mode severity, the failure mode detectability and the failure mode occurrence.
The Generic algorithm to prioritize test case shown in Table 14 use the output results of FMEA and modelbased prioritization algorithm to prioritize test cases and to generate a sorted test suite.

Traceability Matrix
The generated traceability matrix « ReRPNTM » shown in Table 15, identifies links between requirements, potential failure modes, risk priority numbers and Requirement priority number. This matrix tracks a many-to-many relationship -many requirements to many failure modes. One requirement can require multiple failure modes and one failure mode can cover multiple requirements. Typically, the matrix shows the requirements across the top as columns and the associated failure modes down the right side as rows. Each failure mode is associated to a risk priority number (RPN) and each requirement is associated to a requirement priority number (RePN).
Step 5. Construct a generic test model that represent in the same time requirements and failure modes TM.
Step 6. Annotate test model elements with "@Re" for requirements and "@FM" for failure modes.

Discussion
From the fact that model-based testing has emerged as a major research area in academic and industrial, a large number of publications and new approaches are produced in this field. Most of the previous publications give new MBT approaches. For example Graf-Brill and Hermanns (2017), have proposed an approach that use model-based testing to test asynchronous communicating systems. Also, Wang et al. (2017) used model based testing to validate quorum-based systems implemented using the Gorums library through a new approach. On the other hand, some studies give the publications covering supporting techniques for modelling and test generation (Gebizli and Sozer, 2017), integration into industrial practice (Peleska, 2013), taxonomies (Utting et al., 2012), industrial evaluations (Blom et al., 2016), surveys and classification. For example Utting et al. (2012) that have provided taxonomy of model based testing in which the principal aspects of MBT approaches are covered. This paper provides a new model based testing approach to overcome some challenges involved in model based test case prioritization.

Conclusion
In this paper, we have presented a new testing approach to perform test-case prioritization in model based testing way. This makes it possible to efficiently apply prioritization when generation test-cases in model based testing. It consists of generating test cases associated with their priority numbers. In Our approach we have used Failure Mode and Effect Analysis (FMEA) method to analyze failure modes associated to the requirements of the system under test. The generation method is based on a generic test model that represents in the same time requirements and their potential failure modes. This model is annotated with RPNs for potential failure modes to generate test cases with their priority numbers based on failure modes and RePNs for requirements to generate test cases based on requirements. Also, we have introduced a new element named generation directives to direct test generation if it is based on the requirements or failure modes. We concluded that the proposed approach is capable to generate prioritized test cases based on requirements and/or failure modes. As future works, we will extend how to create models that represent requirements and failure modes in testing purpose and how to annotate these models by RePN and RPN. Also, we will detail the generation directives for the automatic test generation and we will adopt an existing test generator to take in consideration our approach' inputs to generate appropriate test cases. And then we will make a case study to illustrate test cases generation by our approach.

Significance Statement
This paper provides a new model based testing approach to overcome some challenges involved in model based test case prioritization. We have developed a new model based testing approach that use Failure Mode and Effect Analysis (FMEA) to prioritize test cases during test generation. The proposed approach aim to generate two types of test cases: Test cases based on requirements and test cases based on potential failure modes.
The generated test cases in both types are associated with their priority number to rank test cases during execution and increase the rate of fault detection. An increased rate of fault detection can give an earlier status of the system under test in order to find and correct bugs as soon as possible.

Acknowledgment
I would like to express here the very thanks to my dissertation advisor, Prof. Dr. Marzak Abdelaziz, University Hassan II, who provided me the opportunity to do such a research in his laboratory.

Author's Contributions
ATIFI Meriem: He proposed this approach of using, he participated in all experiments, coordinated the data-analysis and contributed to the writing of the manuscript. He designed the research plan and organized the study.
MARZAK Abdelaziz: He participated in the design of the research plan and organized the study. He coordinated and validated the research.

Ethics
On behalf of my co-author, we inform you that this article has no ethical issues, we confirm that it is an original work which hasn't been published elsewhere. Each author has personally and actively reads this work before submission and that in no case will there be an ethical issues.