Empirical Investigation of Factors that Hamper Pursuing Software Process Improvement: Analyses of Saudi Practitioners’ Views

: Software product quality is affected by the quality of the process used to develop it. Improving process quality helps software organizations in developing better software products on time and within budget. To achieve these benefits, software organizations are becoming more interested to pursue Software Process Improvement (SPI) initiatives. The failure in pursuing such initiatives is affected by several factors. The literature reports many studies that document different factors impacting the success of SPI initiatives. Very few of these studies have been conducted in the Middle East and particularly in Saudi Arabia. In this study, we report the results of a survey-based empirical study that identify factors that hamper the SPI initiatives conducted by 26 organizations located in Saudi Arabia. A survey has been sent to various organizations. Responses have been collected and analyzed. Results from the literature review and from our survey have been synthesized and presented in this study along with a comparative analysis of similar factors reported in similar studies worldwide. Knowing, prioritizing and understanding these factors can help both researchers and software development organizations avoid them to successfully plan and execute SPI initiatives.


Introduction
During the past two decades, software process improvement (SPI) gained popularity in the software industry as there is a growing consensus that a highquality software process results in a high-quality software product, reducing cost, meeting deadlines and increasing productivity (ISO, 2015;Ashrafi, 2003;Jiang et al., 2004;Pitterman, 2000;Yamamura, 1999). As part of quality management, SPI is a widely used approach to enhance software process in software organizations . Software organizations tailor their software processes to fit each software project and to efficiently develop software. Nowadays, various reference models and standards are available to help in SPI and achieve higher process capability and organizational maturity. This includes ISO 9000 series of standards (ISO, 2015), ISO15504 (SPICE) (ISO, n.d.) and the Capability Maturity Model Integration (CMMI) (CMMI Product Team, 2010). CMMI model focuses on determining the organizational maturity while ISO15504 focuses on determining the process capability.
Despite the flourishing software industry worldwide, a relatively small proportion of the software development organizations has conducted or is committed to processing improvement initiatives. Various reasons that prevent software organizations from pursuing SPI initiatives have been published (Bayona et al., 2012;Dyba, 2005;Niazi, 2012;2015;Niazi et al., 2008;Staples et al., 2007;Staples and Niazi, 2008;Zarour et al., 2015;Zarour, 2009); for instance, the literature shows that software development organizations are facing problems in the implementation of SPI initiatives using maturity models like CMMI (Staples et al., 2007). Moreover, small-sized and medium-sized organizations have expressed many concerning the relevance and applicability of models like CMMI (Clarke and O'Connor, 2013;Pino et al., 2010;Staples et al., 2007). One of the biggest challenges stated is that an SPI program is a high-priced undertaking and organizations need to commit significant resources over a long period (Coleman and O'Connor, 2008). With reference to the SEI appraisal results pubished online (SEI, 2017), we noticed that the number of appraisals, based on CMMI, is still low, as depicted in Table 1. As can be seen from Table 1, Saudi Arabia is one of the countries with few appraisals, although its IT sector is one of the main sectors in the Kingdom. "Despite the increasing importance and need for an empirically tested body of knowledge on different aspects of successfully implementing SPI initiatives, there has been little research carried out to understand difficulties and challenges involved in implementing an SPI program based on assessment models like CMMI in developing countries" . This research is expected to provide SPI practitioners of Saudi Arabia, as well as, their collaborating international parties, with some insights into the practitioners' perception of SPI and factors that can support its implementation in the Saudi market.
The research questions that motivated this research work are: • RQ1: What are the most important factors that hamper an SPI initiative from a practitioners' viewpoint?
• RQ2: How are these factors related to the Practitioners' roles?
• RQ3: How are these factors related to the Organizations' size?
The rest of the paper is organized as follows: Section 2 presents background and related work related to software process improvement and empirical work conducted to evaluate failure factors. Section 3 presents the adopted research methodology and how it is applied. Section 4 presents the results both demographic results and findings related to the failure factors understudy. Section 5 provides a comparison with similar studies. Section 6 discusses the possible limitations and section 7 presents the conclusion and future work.

Background and Related Work
In this dynamic world, the need for new software products and services is growing everyday. Software organizations are striving to meet customers' requirements. To do so, organizations need to improve their software processes continuously to stay competitive. Software process improvement can be conducted either (Gorschek et al., 2006): Top-Down or bottom-up. A top-down approach compares an organization's process with a generally accepted standard process then aims to eliminate the gaps between the existing process and a standard process. Examples of such reference processes include the CMMI and ISO 15504. The bottom-up approach assumes that process change is driven by an organization's goal, characteristics, product attributes and experiences. Change is defined by a local domain instead of a universal set of accepted practices. Bottom-up SPI approaches assume that every development organization must first completely understand its process, products and software characteristics before it can select a set of changes meant to improve its process. Examples of bottom-up approaches include Six Sigma and Quality Improvement Paradigms.
Although many success stories related to SPI implementations have been published, many organizations are still facing problems in achieving the same results (Dyba, 2005;Niazi et al., 2008;. There are many different reasons, presented in the literature, why an SPI initiative succeeds or fails. Some researchers link the success of an SPI initiative to the organization size (e.g., large or SMEs) (Sánchez-Gordón et al., 2016) (Mutahar et al., 2016;Sivashankar et al., 2010). Others propose that some development processes improve the ability for successful SPI, e.g., SPI and agile methodologies (Chetankumar Patel, 2009;Polavarapu and Jami, 2016). Hence "there is a growing interest over the recent years in SPI for SME's and adopting agile principles for SPI" . Alignment of SPI initiatives with business goals is also considered as one of the critical factors for a successful SPI initiative (Vasconcellos et al., 2017). In fact, many studies have discussed the SPI failure factors; a recent summary of SPI critical barriers identified via literature review and empirical studies have been presented in (Niazi et al., 2004).
Adopting any of the SPI approaches do not come without upfront investment. SPI initiatives are usually "costly and improved processes need time to be disseminated, making the impact of SPI hard to measure and justify" (Coleman and O'Connor, 2008;Méndez Fernández and Kuhrmann, 2015), at least on the short term. Accordingly, interested practitioners are reluctant to conduct SPI (Coleman and O'Connor, 2008). Niazi et al. (2005) presented the importance of using an effective strategy to successfully implement SPI. In summary, there is no lack of SPI reference models, standards or implementation approaches (i.e., bottom-up or top-down) but we lack an effective improvement strategy as well as a deeper understanding of its success factors. One of the reported influencing factors of an SPI implementation is the human factors and the impacted individual's perception of SPI (Münch et al., 2012). In this research, we explore the practitioners' perception of SPI and factors that affect conducting improvement initiatives in light of failure factors discussed and summarized in Niazi's et al. (2004). A summary of failure factors is shown in Table 2. Having a deeper understanding of these factors would improve our understanding of the reasons behind the low adoption of SPI initiatives in Saudi organizations.

Research Methodology
For this research, a survey-based empirical study has been designed to understand the failure factors of SPI initiatives specifically in the Saudi market using practitioners' viewpoint ( Fig. 1). The approach adopted to collect responses is an online survey using Google Forms available online at (SPI, 2016).  The SEPG group in the Software Engineering Department of Prince Sultan University has helped in publicizing the survey among practitioners in IT organizations. The team also explained survey questions to ensure the understanding of participants. Practitioners have been contacted via various means, including phone calls, face-to-face meetings and emails. Those who agreed to participate in this study were contacted by email that includes the link to the survey. Participants are all playing various IT roles in the domestic software industry.
Responses were subjected to cross-response examination to validate their authenticity and relevance to our field of study. A total of 60 practitioners were invited to participate in this study. Thirty-four surveys were completed, giving a response rate of around 57%.
The questionnaires were manually reviewed for correctness and completeness to prevent any irrelevant entries from being included in the results. Eight surveys were excluded due to incomplete or irrelevant answers. We ended up with 26 valid surveys.
According to the number of valid surveys (n = 26), we cannot claim that the collected sample is statistically representative for the local IT market size of the Kingdom. A truly representative sample is impossible to attain and the researcher should try to remove as much of the sample bias as possible in order to make the sample fairly representative (Coolican, 2009). But what we are sampling here is the practitioners to represent different kinds of IT organizations, where we believe twenty-six surveys are suitable sample size according to (Krejcie and Morgan, 1970). Note that we are not surveying certain organization that adopted SPI but we survey practitioners' perception of SPI, hence, we were committed to collect data from practitioners with various roles that includes developers, requirement engineers, designers and quality assurance representatives, team leader/ project manager and senior managers. Accordingly, we can say that although the sample is not random it is a convenient sample.
The frequency analysis method is used to group the data into frequency tables to facilitate reading and analyzing descriptive information. This is useful in comparing and contrasting within groups of variables or across groups of variables. To analyze the failure factors, the occurrence of each factor in each questionnaire was counted and the relative importance of each factor was identified by comparing the occurrences of one factor against the occurrence of other factors.

Demographic Results
The survey started by asking some demographic questions to understand the background of the participants and their organization nature. The following are the findings of the demographic questions: • Participants' Roles: Participants were asked to specify their main role in the IT organization. Roles can be either project manager/team leader or technical member. 68% of the respondents were project managers/team leaders while 32% were technical members (Fig. 2a) • Participants' IT-related activities: Participants play various roles in their IT organization. Figure 2b summarizes the various roles played by the survey respondents • Organizations' SPI initiative and which Model is used? The participants have been asked whether they know any popular SPI models; 42% said yes, they know some SPI models, while 58 said that they do not know any SPI model (Fig. 3c). Then the participants were asked if their organizations have conducted any SPI initiative. 54% said yes their organization have conducted an SPI initiative, 19% said no while 27% do not know if their organizations did conduct any SPI initiative (Fig. 3d) • Awareness of SPI Initiatives in the Organization. For those who said that their organizations have adopted SPI initiatives, we asked them if they know what SPI model(s) are used. 81% said that they do not know which model is used exactly while 19% indicated that they do know the SPI model used (Fig. 3e) The adopted SPI models were CMMI with adoption rate of 60% while ITIL and Kanban were adopted equally with 20% adoption rate for each (Fig. 3f)

Survey Findings
Factors That Mostly Hamper the SPI Initiatives from Practitioners' Viewpoint (RQ1) Table 3 lists the SPI failure factors from the viewpoint of 26 practitioners who participated in the study. 92% of Practitioners agreed on all the failure factors with a rating more than 50%. The two most influential failure factors are the ability to change the mind-set of management and technical staff and the lack of sponsorship. The second most significant failure factors are lack of formal methodology and time pressure. In the 'Negative' column, values are less than 50%. The same thing applies to the 'Neutral' column which means the practitioners agreed on the importance of the failure factors. Interestingly, practitioners do not see SPI imitative as obstacles that get in the way of real work as they rated this failure factor the least influential factor.

Factors Identified based on Participants' Roles (RQ2)
To analyze the SPI perception of various practitioners' groups, the practitioners have been divided into two main groups: • Technical members: This includes developers, requirement engineers, designers and quality people • Managers: This includes team leader/project manager and senior managers Table 4 lists the failure factors and ratings of both groups. Managers rated the failure factor 'Time pressure' in the first place and in the second place come the 'Ability to change the mind-set of the management and technical staff', 'Lack of formal methodology' and 'Lack of sponsorship'.    Three failure factors are rated in the first place by technical team members that include: 'Ability to change the mind-set of the management and technical staff', 'Lack of sponsorship' and 'Lack of support'. In the second place, technical members rated two failure factors that include: 'inexperienced staff/lack of knowledge' and 'Lack of formal methodology'.

Fig. 3: Demographic Data II (a) Organization Size (b) Participants Years of Experience (c) Do you know any SPI model (d) Does your company adopted SPI Initiative (e) If your company adopts SPI Model, do you know which model is used? (f) Adopted SPI Models in the Organizations
To analyze the significant difference in the responses of technical members and managers to the survey questions which are of ordinal type (Likert scale), we used Fisher exact test with (α = 0.05). As shown in Table 4, none of the challenges has a p-value <0.05 which means there is no significant difference in any of the specified failure factors across the two groups of technical members and managers.   Large (13) Medium (2) Small (1) Table 5 lists the failure factors and ratings based on organization size grouped into three groups: Large consists of 13 organizations; medium consists of 2 organizations and small consists of 1 organization. The ten remaining organizations did not specify their organizations' size hence they have been excluded from this analysis. Two failure factors are rated in the first place by practitioners from large organizations, namely: 'Lack of formal methodology' and 'Lack of sponsorship'. Practitioners from medium size organizations rated five failure factors in the first place, namely: 'Lack of resources', 'Lack of sponsorship', 'Lack of support', 'Paperwork required/formal procedures' and 'SPI gets in the way of real work'. Practitioners from small size organizations rated eight failure factors in the first place, namely: 'Inexperienced staff/lack of knowledge', 'Lack of formal methodology', 'Lack of support', 'Negative/bad experience in SPI', Organizational politics', 'Paperwork required/formal procedures', 'SPI gets in the way of real work' and 'Time pressure'.
This means that participants from small and mediumsized organizations are likely with a true null hypothesis (Ability to change the mindset of management and technical staff is not a failure factor). On the other hand, when calculating Fisher exact test for samples from large and medium organizations, the p-value was 0.0019 and from large and small organizations, the p-value was 0.0049. This means that participants from large organizations and small or medium size organizations are unlikely with a true null hypothesis. That is, practitioners from large organization see that 'Ability to changing the mindset of management and technical staff' is a failure factor while the practitioners from medium and small organizations do not see this as a failure factor! In other words, the mindset of management and technical staff in small and medium-sized organizations is more open and flexible in accepting new SPI initiatives than those in large organizations despite other factors.

Comparison with Similar Studies
Another interesting dimension in this study is the comparison with similar published studies worldwide. Table 6 presents a summary of failure factors identified by Saudi practitioners compared to similar factors identified by Australian, English and Vietnamese practitioners as depicted in   (Baddoo and Hall, 2003) and (Niazi and Ali Babar, 2007) respectively. Niazi et al. (2008) have studied SPI implementation critical barriers in twenty-six Australian software organizations. In, (Niazi and Ali Babar, 2007) conducted an empirical study with 23 Vietnamese software practitioners to determine SPI de-motivators. Baddoo and Hul (2003) have conducted a study to identify de-motivators of software development practitioners in the United Kingdom.
By analyzing the de-motivators that come in the first place for each sample, we can see the following, Saudi practitioners rated the 'ability to change the mindset of management and technical staff' in the first place, while English practitioners rated this failure factor in the sixth place, Vietnamese rated it in the second place and for Australian, this factor came in the seventh place. It seems that management and technical staff in the developed countries (the UK and Australia) are more open to adopt improvements and change accordingly than in developing countries (Saudi Arabia and Vietnam). I wonder whether it is the mindset of managers and technical staff or the organization's mindset that resist change. This thought needs further studies to reveal whose mindset is difficult to change.
For English practitioner, time pressure is the main barrier to successful SPI initiatives. Saudis rated this in the second place while Vietnamese and Australians rated it in the fifth place. It seems that English practitioners are working under pressure most of the time that affects badly new initiatives (e.g., SPI initiatives) followed by Saudis. Australians and Vietnamese are more released from this pressure. The question here is what makes practitioners in certain countries work under time pressure? Is it related to some constraints that are not under practitioners' control (e.g., time or resources constraints), is it due to unforeseen problems and challenges, do practitioners have insufficient knowledge, work on difficult tasks, or do the managers lack planning and management skills? More research and root-cause analysis are needed to uncover time-pressure related issues.
Vietnamese practitioners suffer more from lack of resources followed by English practitioners then Australians and finally Saudis. This can be referred to the fact that Vietnam is a developing country that is still suffering from lack of resources , Saudi Arabia is a developing country as well but with an abundance of resources.
Australians stressed the need to secure necessary support to SPI initiatives than practitioners in other countries. They also considered organizational politics as the main barrier when adopting SPI initiatives.
To identify the statistical dependence between the ranks of the different samples (Australian, Vietnamese, English compared to Saudi practitioners), the Spearman's rank-order correlation was used. Spearman's correlation coefficient gives the linear dependence between two organizations (Saudi Sample and each other sample) with 1 being a total linear dependency. Spearman's correlation coefficient values ranging from −1 to +1.    As shown in Table 7, the Spearman's correlation coefficient, Rs, is 0.0663, 0.2048 and 0.1702 respectively. This shows a weak positive correlation between the results obtained from the two samples. The obtained results are statistically insignificant (p = 0.8768, 0.05021 and 0.5783 respectively).

Limitations
The failure factors used in this research have been studied and identified in the SPI literature. Hence, the failure factors used in this research are collected from few but highly ranked published articles which, in their turn, have conducted a formal literature review to identify these factors.
We believe that the selected papers cover the most relevant published literature in the SPI domain.
Regarding the conducted empirical study using the questionnaire survey, one of the possible threats related to construct validity is concerned with practitioners' interpretation of each factor under study. We solve this issue by providing contact details for the author and his SEPG team to answer any question of any participants in the study.
Although the survey is published online, many of the participants have been met face-to-face and survey questions have been explained to them before they start filling it. We do not have any evidence that proves this limitation as none of the participants reported it as a problem. With regard to the internal validity, the measurement scale used is based on a clear Likert scale and the questions are based on failure factors that are clearly documented in the literature. SEPG Members have answered the survey, as a pilot study, to verify its clearness and preciseness before submitting it to practitioners. Moreover, the generalization of the results to other situations and people undergoes further research; hence we cannot generalize it for the time being and this makes external validity in process.

Conclusion and Future Work
In this research, we have identified the common failure factors documented in the literature for adopting SPI initiatives. These failure factors have been formally studied, reviewed and documented in the literature.
We analyzed the experiences, opinions and views of practitioners related to these factors to identify which factors that have a negative impact on the implementation of SPI programs. We believe that focusing on these factors offers SPI practitioners the opportunity for implementing practices that have an impact on the SPI implementation process.
We have also compared the findings of our empirical study in Saudi Arabia with similar studies in the litterateur and found that there are both similarities and differences between practitioners in rating failure factors. Our results also show that different groups of practitioners are aware of what is imperative for the implementation of SPI programs.
This work is to be complemented with another ongoing research that would document the experience of Saudi organizations that have conducted SPI initiatives (mainly based on CMMI) and have a closer look on the real barriers faced while implementing SPI programs.