Evidence-Based Study on Performance Evaluation of Social Education Institutions with Balanced Scorecard

Problem statement: Based on the concept of Balanced Scorecard (BSC), this study was intended to assist the establishment of performance evaluation system for social education institution s in transforming phase. Approach: An evidence-based analysis was processed to examine the objectivity of the indicators and the weights of ke y factors of the current performance evaluation. Results: Based on the concept of BSC, literature analysis, expert interviews and Analytic Hierarchy Process (AHP), this study investigated the propriet y of the performance management indicators currently used by social education institutions in transforming phase, with the intention to establish a more objective and practical system. Conclusion: The advantage of this study it can provide a practical framework for will construct and carry ou t the management system of the performance evaluation for other social education institutions i the future. Key word: Balanced Scorecard (BSC), Analytic Hierarchy Proce ss (AHP), performance evaluation


INTRODUCTION
Facing rapid change of global environment since 1980s, governments of all nations have been involving in reformation and innovation. For example, American President Clinton passed Government Performance and Result Act of 1993 (GPRA) and took government performance evaluation toward Legislation level. British government also proposed Citizen's Charter in 1991 for reformation. All nations in the process of reformation aimed to establish a performance evaluation system. Kaplan and Norton (1996) indicated to assess whether any institution run by government and/or other nonprofit organizations was cost effective should base on the satisfaction of the electorate and/or the respective sponsors. The organization should orient its goals on customers and/or the electorate. Financial factors might take the role of restriction and promotion, but they should not be the main target.
The scholars in our nation also responded to such growing competition and devoted themselves into research of Balanced Scorecard (BSC). Most studies were done on commercial enterprises rather than on government organizations and social education institutions. They focused mostly on presentation of processing, experience sharing, or simulating constitution of performance evaluation indicators. For example, the study of Chen et al. (2006) on relation between key elements to measure Chinese and Japanese hospital performance; the study of Coop (2006) on Balanced Scorecard application in the mental health service for a New Zealand; the study of Kocakulah and Austill (2007) applied BSC in the health care industry; the study of Parkinson et al. (2007) indicated a critical review of financial measures as reported in the hospital BSC; the study of Patel et al. (2008) proposed balancing the NHS BSC; the research of Norreklit (2000) on exploration the balance on the balanced scorecard a critical analysis of some of its assumptions; the study of Cleverley and Cleverley (2005) on the discussion of using financial metrics to improve performance in Scorecards and dashboard; the study of Beard (2009) on the research and successful application of the Balanced Scorecard in Higher Education; the study of Kettunen (2004) on the evaluation of regional development in higher education based on the concept of Balanced Scorecard; the research of Andon et al. (2008) on the discussion of using the balanced scorecard in slogans, seduction and State of Play.
However, the issue of how to determine the proper weights of key factors for performance evaluation directly affected the future development, resources balance and the equitable performance evaluation for each division which influenced the morale of an organization. Some researchers began to notice such issue; Analytic Hierarchy Process (AHP) could provide some insight for the performance evaluation indicators and the weights of key factors yielded from Balanced Scorecard (BSC); such as Ramanathan (2001) researched a note on the use of the analytic hierarchy process for environmental impact assessment. Utilizing the concept of Balanced Scorecard and application of Analytic Hierarchy Process to select objective performance indicators and the weights of key factors, our study discussed how social education institutions could rise above the change in demands, the wrestle of political parties and competitive environment to focus on development and management policy, to make alignment with organization resources, so to transform possible crisis into great opportunities.

The introduction of social education institution in transforming phase:
The goal of general social education institution, according to the government Educational Policy Write Paper since 1988's, social education act, life education act and other relating policy and regulations, was to constructively promote social education. Those organizations regularly held various life education activities and production exhibitions, their internal structure was generally divided into business, administration and preservation divisions and an evaluation committee composed by one senior officer, 2/3 of directors and 1/3 of experts and specialists was also set up. They worked together to establish the perspectives, indicators and the weights of key factors of performance evaluation for the upcoming organization fund in order to reach the expected goals.
For most social education institutions, the general goal often emphasized the promotion of life education services. They seldom focused on the strength and weakness of their competition. Because most were nonprofit organizations and their expenditure was covered by government budget, the essence of performance evaluation was often overlooked. Since most institutions were not previously confronted with immediate survival crisis, they lacked efficiency in operation. However, facing internal environmental change and government's financial predicament, transformation of social education institutions seemed more necessary. In addition to search for more resources and to execute organization foundation, social education institutions had to compete with many universities, community colleges and open universities which began to set up and accommodate continuing education. Some social education institutions were forced to face the adversity of not having enough students for classes and the poor quality of exhibition. Thus, how to elevate cost-effective output value and efficient services had become the top priority for social education institution in transforming phase. Traditionally, evaluation of output value and service performance was assessed by quantitative financial statements and focused on the outcome instead of process. Even the social education institutions of Public Affair division in government aimed to promote services and perceived the annual performance efficiency solely relying on revenue statistics. To encourage organizations for elevating administration efficiency, the Executive Yuan has been advocating a plan under the title of "Carry out a performance bonus project for Administration Yuan and all levels administration organization" since 2003, hoped to establish a system in which financial perspective was only one of the weighing factors in performance evaluation.
Under current educational framework, the social education institutions not only bear the mission of community education, but also serve as locations for collection, exhibition, tourism, repose and diplomacy. To consolidate the institutions in providing better service and increasing revenue, one objective and reflective performance evaluation system is in need. Since the government guarantees direct flight cross straits and more Mainland visitors (President Ma's presidential speech of 2008), the role social education institutions play becomes more crucial. The Balanced Scorecard (BSC) was publicized by Kaplan and Norton (2004) and named by Fortune Magazine as the most influential theory of the 20th Century. It is comprised of four perspectives: Labeled as financial perspective, Customer perspective, Internal Business Processes perspective and Learning and Growth perspective, respectively. BSC is often applied in enterprise performance indicators to determine the key measures; Analytic Hierarchy Process (AHP) proposed by Saaty (1977) is also utilized to obtain the relative weights of the key factors in BSC. In addition to have organization employees realize enterprise's vision, BSC with AHP also facilitate front officers and policy makers to follow up the outcome of policy.
Purpose: Conventional performance management or strategy management in the past did contribute much in staff management and goal fulfillment for organizations.
Among various methods in assessing performance, many managers came to one consensus: that earnings based on financial statistics could evaluate only the outcome of policy made in the past, not that of future development. Robert G. Eccles proposed that one should not search enterprise performance indicators solely from financial statistics; the non-financial indicators such as quality, customer satisfaction, innovation and learning and market share could better reflect operation condition and growing vision of the enterprise from his building public trust book. Stepping into the era of knowledge economy, intangible assets seem more important than the tangible ones. Innovation, public praise, reputation and employee morale of an enterprise all are intangible, yet they motivate excellent quality and production. Lately organization managers are gradually changing their evaluation system. In order to intensify new competitive strategy, they tend to invest more in innovation and services and incorporate more nonfinancial indicators into performance assessment. Kaplan and Norton (1992) introduced in the concept of Balanced Scorecard (BSC) and should be a component of a strategic management system that links the entity mission, core values for the future with strategies. They believed in addition to financial perspective, Customer perspective, internal business perspective and Learning and Growth perspective should also be integrated to seek enterprise performance of long-term and short-term goals, financial and non-financial measures, lagged and leading indicators and external and internal performance perspectives. Porter (2001;2008; studied the relationship of strategy and internet; Kaplan and Norton (2004) proposed the strategy and believed that while pursuing accomplishment and performance, an enterprise should also accumulate more strength, acquire more intangible assets and develop greater vision. Many performance management strategies existed, but whether the chosen strategy brought benefit for the organization and was well taken by all employees should be the main concern for the directorial. As for multi-functions of social education institution nowadays, the government wishes for more exhibitions first and then related social education and tourism second. However, due to gradual shortage of financial resources, the institutions cannot fulfill the expectation and are in need for transformation.
The elevation of operation performance for all enterprises has become urgent under the current competitive environment, especially that of social education institutions. By application of Balanced Scorecard (BSC) in performance management strategy, substantial perspective indicators are selected and examined. So managers can clearly detect and correct defects and intensify advantages. While cooperating to government policies, social education institutions can display their characteristics and functions in uplifting art and culture level and becoming international tourism attraction. The application of BSC is anticipated to be extended in the future into performance evaluations of all divisions and each individual employee. However, Leung et al. (2006) stated that too many indicators in BSC would deplete its function. Chan (2006) studied an analytic hierarchy framework for evaluating balanced scorecards of healthcare organizations; Chan (2009) applied AHP framework for evaluating BSC of healthcare organizations. The performance evaluation which used Analytic Hierarchy Process (AHP) application to measure the weights of the key factors was highly correlated to the return-on-investment of stock market, according to the research result Wyatt (2004) proposed that scorecards, dashboards and KPIs keys to integrated performance measurement.
The population of our study included all employees (evaluation committee, senior officers and other staff) and the volunteer workers whose working experiences were three years and longer in the social education institutions in transforming phase. Based on the concept of four perspectives of Balanced Scorecard (BSC), our study made alignment with policies for financial resources, core value segment of services, internal process value chain of customers and management group, internal skills and capabilities, technology foundation and motivation to learning and growth factors and connected all elements of four perspectives using Strategy Map. Analytic Hierarchy Process (AHP) was applied to acquire the weights of performance indicators derived from BSC. Wilcoxon signed-rank test was put to detect any significant difference between the weights of indicators selected by current evaluation committee and those derived from AHP.

MATERIALS AND METHODS
This study was intended to establish an objective performance evaluation system for social educational institutions in transforming phase. The research process was as the following: , current performance evaluation perspectives and relating indicators. Preliminary coordination and analysis were constructed via the expert interviews, the perspectives and related indicators of BSC of social education institutions in transforming phase were then mapped out. Analytic Hierarchy Process (AHP) was applied to the above data, questionnaires was designed with Likert's 7 point measure form. • One hundred questionnaires were distributed: Seventy to the personnel of social education institutions (including evaluation committee members, senior personnel and officers), thirty to the volunteers whose working experiences were longer than five years

Application of Analytic Hierarchy Process (AHP)
software: AHP built a clear multi-level frame with elements relating to all aspects of the targeted problem. The numerical weight was derived for each element of the hierarchy with mathematical method of pair-wise comparisons. The decision makers could use the data about the elements and cut down the complexity of making judgments. The weight of each element was obtained based on the characteristic vector, so it contained certified mechanism of consistency.
Wilcoxon signed-rank test: Wilcoxon signed-rank test, according to Saaty and Vargas (2001), was a nonparametric statistic method involving comparisons of differences between two related samples. In addition to positive and negative deviations of paired samples, ranks of deviations were also taken into consideration. The test was quite efficient. Present study applied Wilcoxon signed-rank test to detect any significant difference between the weighing indicators selected by current evaluation committee of social education institutions and those derived from AHP.

RESULTS AND DISCUSSION
Combing literature analysis, expert interviews and the perspectives and indicators currently used by social education institutions in transforming phase, our study set up the four perspectives and their related indicators based on the concept of Balanced Scorecard (BSC). And by the application of Analytic Hierarchy Process (AHP), each perspective, strategy goal and the weight of performance indicators were established for future assessment.

Establishment of Balanced Scorecard (BSC):
To establish Balanced Scorecard (BSC) for social education institutions, the performance evaluation along with related literature reviews (Kaplan and Norton, 2004). Followed by interviews with senior officers of two institutions with similar characters, a draft BSC was accomplished. Financial perspective included budget is made and achieved actually and surplus is increasing; Internal Business perspective included performance goal is achieved, carry out a quality service and the internal administration process is high efficiency; Learning and Growth perspective included staff core skills shall be promoted, organization atmosphere shall be aggressive, employee's discipline is strictly and e-administration. The performance evaluation of social education institutions in transforming phase lacked Customer perspective. In 2007 higher authority hired experts to assess organization operation. The result coincided with the above outcome. Social education institutions belonged to government division and were located within city center; they not only promoted social education and activities, but bore the function of exhibitions and tourism. The customer's social education institutions served exceeded two million people annually, including local residents, general public and international visitors. Thus, customer perspective became significantly crucial.

The calculated result of Analytic Hierarchy Process (AHP):
The survey candidates of AHP, in addition to evaluation committee members, directors, other personnel and volunteers whose working experiences were three years or longer, amounted to the total of one hundred questionnaires. Ninety-one was returned (the return rate was 91%); of which 75 questionnaires (84%) were staff members whose working experiences were longer than eight years. Such outcome matched the Consistency Index of AHP (CR< = 0.1). There were 91 valid questionnaires and 82% valid return questionnaires. As for validity, the perspectives and indicators of BSC used in questionnaire were based on literature reviews, expert interviews and the current performance evaluation system of transforming social education institutions. They were then tested and corrected by senior officers of performance management.
After applied AHP, the weight of customer perspective of BSC was 31.2%, followed by financial perspective (26.6%), Internal Business Process perspective (23.1%) and Learning and Growth perspective (19.1%), respectively. The top five weights were student return rate (28.3%), visitor return rate (26.9%), visitor complaints (11.7%), Taipei city international touring site excellent grading (10.1%) and budget achieved rate (57.3%). Four belonged to Customer perspective and only one of financial perspective.
The relation of weights of indicators of current performance management system and those of Analytic Hierarchy Process (AHP): The performance management system of current social education institutions in transforming phase valued more of Financial perspective and Internal Business Process perspective (the weight of each perspective was both 35%) and lacked emphasis on customer perspective. The difference from that based on Balanced Scorecard (BSC) could be distinguished once AHP was applied. To explore the significant discrepancy between the two, our study utilized Wilcoxon signed-rank test and the result showed P = 0.0135<0.05. This outcome suggested that significant difference existed between the weighing indicators of current performance management system and those of AHP. It presented the possibility that the weights of performance evaluation indicators derived by Evaluation Committee and senior officers might be overly subjective.
Since Balanced Scorecard (BSC) began to rise and develop in 1990s, most enterprises and government organizations recognize and approve its function. BSC is considered the latest performance management system because it possesses the function of evaluating performance, management process and management plan. BSC can guide, control and balance the development of an organization with its self-determined feedback mechanism. Our study found that the perspectives and indicators of performance evaluation of any organization could not be derived solely from the opinion of few leading officers. To exclude any bias and subjectivity, an efficient performance evaluation was crucial. Based on the concept of BSC in establishing a performance system, experts of BSC, organization chiefs, senior officers, other personnel and volunteers were recruited. Their comments and opinions were summed up, along with literature reviews, interviews, conferences and other objective methods to determine performance perspectives and measuring indicators. Questionnaires and Analytic Hierarchy Process (AHP) were then applied to the data for assessing the priority of perspectives and the weights of indicators in each perspective. With recognition and approval of organization and its employees, the final performance management system could then be taken effect.

CONCLUSION
For most government organizations, the indicator of performance strategy still focused on financial perspective, which limited the performance efficiency and benefit. Social education institutions in transforming phase currently were confronted with the predicament of fundraising. If performance system for social education institutions in transforming phase was constructed solely by evaluation committee and chiefs, the subjective judgment would affect the efficiency and blur detection of problems. Thus, the incorporation of an objective, effective and practical performance evaluation based on Balanced Scorecard would greatly benefit future operation, Furthermore, with the application of Analytic Hierarchy Process, the perspectives, goals and weighing indicators of performance evaluation system were clearly established. It was advised that the same system be applied to other institutions with similar character before execution for reference and future improvement.