Model-Driven Framework for Evaluating Learning Outcomes Process

Corresponding Author: Mohammad Alhaj Department of Computer Engineering, Al-Ahliyya Amman University, Amman, Jordan Email: m.alhaj@ammanu.edu.jo Abstract: Evaluating learning outcomes in academic institution can be complex and challenging. Several quantitative and qualitative assessment approaches have been adopted to enhance the process of managing, measuring and visualizing the learning outcomes. The difficulty of implementing and analyzing the evaluation process is mainly caused by the nature of the raw data used in assessment. The data is usually unstructured, complex, text-heavy and collected in high volumes. It may also be extracted from heterogeneous platforms and require privileged accessibility. Using paper-based assessment, such as rubric, in complex evaluation process may cause error prone, confusion in analyzing the learning outcomes and subject to different interpretations of the assessment by academic constituencies. In this study, we propose a model-driven framework for evaluation process of the learning outcomes. The framework has four activities: The data collection and data processing activities are used to extract complex data into a useful information for assessment. The model-driven assessment activity is used to generate and analyze goal models of the learning outcomes in a formal way and allows the assessment at different level of academic institutions. Finally, the evaluation reporting activity is used to generate reports that summarizes the institutional status, metrics and real-time data in a form visual object. A prototype implementation of the framework is evaluated using a case study of an ongoing project at AlAhliyya Amman University.


Introduction
Educational institutions are adopting different learning assessment methods and techniques to evaluate and improve student learning outcomes. The evaluation of the learning outcomes is an ongoing enhancement process conducted at the educational institutions and begins when student enters the college until the time of graduation. The evaluation process can be performed at different educational levels depending on the institution structure; starting from the course or the curriculum level and ending up to the program or the college level when a clustered feedback is needed.
A continuous evaluation is held every academic semester where student's achievement and feedback are used to measure the performance of the learning outcomes. Figure 1 illustrates a data model of the education domain (Alhaj, 2019). It represents the learning outcomes and objectives that determines the performance road map to measure the institution situation and progress. Any academic institution have a mission that comes at the top of the model and defines the guidance to the purposes of an institution. There are three mission statements: The institutional mission statement is derived from the institutional vision and describes what is needed to be done to meet the vision; and then the faculty mission and program mission are stemmed from it (Ganu, 2013).
A multi-level structure with three levels, i.e., Course, Curriculum, Program is included in the evaluation process of the SOs, PEOs and Mission. The evaluation process is used to systematically review the input metrics of the SOs with respect to the PEOs. A cyclic feedback principle is used where the evaluation process is applied on each academic program at every semester (Kuo and Hwang, 2014). A group of selective courses in the curricula are used to measure the accomplishment of the enrolled students based on the SOs of each individual course. Also, measuring the impact of the clustered SOs results of selected courses on the PEOs. At the end of each semester, a recommendation for improvement is formulated, applied to the system and tested by the participated constituencies. Typically, all constituencies must participate actively at all levels and the interaction between them will lead to a more efficient assessment. However, this may not be always the case, since the evaluation may not be maintained with ease and consistency all the times.
The following are the five phases of cyclic evaluation process as described in Fig. 2:  Planning phase: It is used to outline the necessary assessment elements, such as the PEOs, the SOs of each course in the curriculum and the linking between them, the target students, the groups responsible for collecting data, the assessment tools for each level and collection rate of the assessment  Assessment phase: For selective courses, multiple direct/indirect assessment tools are used to study and analyze the academic records of the enrolled students with respect to the SOs and PEOs  Committee Evaluation: The evaluation is performed typically by the academic committee of the program who performed the assessment. During the evaluation, recommendations for program improvement are made  Adoption phase: Using the results produced by the assessment tools, program committees prepare their recommendations for program improvement. The recommendations may have either a short-term effect that is refined to meet the educational institution policies and bylaws or long-term effect that need a further discussing from the deanship committees  Implementation phase: The approved recommendations are then implemented by the program members of the program and constituencies are notified with the major program improvements during the general forums. The process returns to the planning phase and repeats for another session

Problems in Current Practices for Evaluation Process of Learning Outcome
In the current practices, the process of evaluating learning outcomes in academic institutions is still relatively immature. The difficulty of implementing and analyzing the evaluation process is mainly caused by the nature of the raw data used in assessment, that is student's achievement and feedback. The generated raw data by the direct/indirect assessment tools is usually not organized in a pre-defined manner (unstructured data). It is also complex, text-heavy and collected in high volumes. The raw data may also be handled by heterogenous platforms and require privileged accessibility when it comes from different management-levels of the institution. Due to the nature of the raw data, it is difficult to implement an evaluation process without processing the raw data through steps of cleaning up, classification and segmentation. There is also a need to have a pre-defined data model used to describe the model elements and incorporate them with the performance metrics used for assessment in the evaluation process of learning outcomes.
The proper solution of simplifying the implementation of the evaluation process is to create an integrated hierarchical structure that separates the evaluation process into interoperable activities. The structure simplifies implementing the evaluation process, handling complex data, augmenting performance metrics with their relevant complements. It also supports establishing clear body for work and management and provides a clear line of communication between different constituencies and management-level of an institution.
In this study, we proposed a model-driven framework for evaluation process of the learning outcomes. The framework consists of four activities: Data collection, data processing, model-driven assessment and evaluation reporting. The framework introduces the data processing in order to extract meaningful information in the evaluation process. It also uses Goal-oriented Requirement Language (GRL) to model the elements of the assessment; and provides Key Performance Indicators (KPIs) for the quantitative measures of SOs compliance with respect to the PEOs during the periodic evaluation process. This will improve the evaluation process and assessing the learning outcomes and objectives in formalized manner and reduce the complication and ambiguity of the analysis. Finally, evaluation reports are generated to summarize the institutional status, metrics and real-time data in a form visual object. The reports include charts and tables that communicate how well the students' achievement and feedback is contributing to learning outcomes and objectives.
The paper extends and elaborates further details on earlier research results presented at the conference in (Alhaj, 2019). It is organized as follows: Section 2 presents the background and related work; section 3 presents the proposed framework of learning outcomes evaluation process; section 4 presents framework prototype, section 5 presents a case study at the computer Engineering Depart. Of AAU; section 6 presents an evaluation of the proposed framework section 7 conclusions and future work.

Background and Related Work
Several researchers are using goal-oriented languages to model and analyze the legal compliance of business processes (Ghanavati et al., 2011), Business Process Management (BPM) compliance (Shamsaei et al., 2011) and enterprise quality assurance (Alhaj et al., 2017). Goal-oriented modeling is used in requirements engineering activities to capture the model elements of business goals and objectives, actors, processes, connections between model elements and the quantitative/qualitative performance metrics on different quality aspects. Analyzing such models improves the decision-making process and compliance with the business goals, provides formalized structure and reduces the ambiguity in user requirements. User Requirements Notation (URN) (Amyot and Mussbacher, 2011) is a standard modeling notation that provides the Goal-oriented Requirement Language (GRL) to model business goals and the Use Case Map (UCM) to model business scenarios. GRL allows the designers to model intentions (e.g., goals, resources, indicators) their breakdown structure (e.g., sub goals, stubs), connection types (e.g., decomposition, reuse) and the related stakeholders (actors, systems, objects). An eclipse graphical editor plugin, called (jUCMNav, 2017), is used to create GRL goal modeling and UCM scenario modeling. jUCMNav facilitates creating, modeling and enhancing complex GRL models. It supports themes that utilizes strategies using several analysis algorithms, supports executing, visualizing analysis results and generating documents.
The recent researchers use different learning assessment and feedback approaches that aims to evaluate the student learning outcomes and educational objectives of an academic institution. The Canadian Engineering Accreditation Board in (CEAB, 2017) introduce the Graduate Attributes (GA) which defines knowledge, qualities and skills should be provided by the university community to its students during their study. Csorba et al. (2013) extends the graduate attributes/sub-attributes and connects them with the performance metrics of rubric. While (Ostafichuk, 2012) defines an outcome-based assessment process with six activities that aims to provide a cyclic assessment and analysis of the program and course improvements. McGourty et al. (1998) presents a five step continuous improvement process for developing an integrated assessment program that starts from defining educational objectives up to applying measuring method. Felder and Brent (2003) propose formalized rules for course learning objectives and their assessments to address an ABET outcomes 3a-3k. They also propose a technique to integrate the activities of the course and program levels to meet the ABET engineering criteria. Yue (2007) proposes a coursebased approach to correlate learning outcome objectives with ABET accreditation standards. He also describes three course assessment tools that supports the approach. While (Abbadeni et al., 2013) reviews the existing processes for evaluating PEO's at the Electrical and Computer Engineering Department (ECE) of Lafayette College. The author is then comparing the implemented processes with other institutions and demonstrates the lessons learned of evaluating the PEO's at Lafayette's ECE Department.
There are also several tools and applications used in the evaluation process of learning outcomes in today's market. Researchers in (George et al., 2016) present a developed tool that supports performance management of graduate attributes. The tool is used to simplify data collection, improves visualization of results and provides a flexibly to integrate internal indicators or external indicators. While researchers in (Kaupp et al., 2013) compare between eleven different software tools supporting outcomes based assessment as part of a continuous improvement process. A comparison is performed based on multiple criteria, such as learning content management, rubric-based assessment, learning outcomes, kind of reporting and pricing. A summary of the comparison based on the evaluation criteria shows that the common weaknesses between the tools varies into lack of integration between the modules, limitations of the outcomes analytic capabilities and evaluation reporting and less efficient assessment tools.
In summary, it is clear that the researches above are describing many of the features that are similar to our work. However, all of the above proposed assessments are paper-based, such as Rubric, where documents are used in the evaluation process of learning outcomes. We developed an integrated hierarchical structure framework that separates the evaluation process into interoperable activities.
The hierarchical structure simplifies implementing the evaluation process, handling complex data, augmenting performance metrics with their relevant complements. The framework also supports a model-driven assessment using GRL goal modeling in evaluating the learning outcomes. This will improve the evaluation process and assessing the learning outcomes and objectives in formalized manner and reduce the complication and ambiguity of the analysis. The framework also supports generating executive and performance reports that help decision maker in monitoring the institution progress and data necessary for future forecasting.

Proposed Framework of Learning Outcomes Evaluation Process
In this section, we present our proposed framework, as in Fig. 3. The framework describes a bottom-up process where four activities are defined: Data collection, Data processing, Model-driven assessment and Evaluation reporting.

Data Collection
In data collection, the academic records are gathered as raw data from different resources. The resources can be either in the form of natural language, such surveys, interviews and academic data from the portal; or structured format, such as databases and spreadsheets. Surveys, interview and discussion groups are a kind of paper-based or webbased questionnaire reports. They reflect the satisfaction of the constituencies to the provided services at an institution. Different kind of surveys are used for that purpose with a motive of assessing the institution's regulations, polices and activities with respect to its Missions, PEOs and SOs. Common surveys used in the academy are: Trend surveys, panel surveys, cohort surveys (Alhaj et al., 2020).
The database is used to persist high volume of academic records, such as students' profiles, achievements, grades, financial records and HR in structural manner. The academic portal contains the various details that can be used in data collections, such as statements of learning outcomes and objectives, institutional structure, academic and department objectives, details. Of academic instructors and enrolled students and alumni.

Data Processing
The raw data in data processing is organized and analyzed to obtain a meaningful information for assessment. The purpose of data processing is providing a better analysis and presentation of the data in order to make a precise decisions and increase productivity and profits. There are several methods of analyzing data:  Cohort data where common characteristics of data records are analyzed, e.g., analyzing student's records based on gender or age  Spatial data where data records are identified based on the physical locations or regions, e.g., analyzing student's records of different programs within the same faculty  Temporal data where data records are identified based on the periods of time, e.g., analyzing student's records of different years and semester within the same program  Multi-level management data where data are identified based on the hierarchal structure of the management in an institution, e.g., analyzing student's records at the course, curriculum, program department and faculty levels

Model-Driven Assessment
A model-based language is defined to assess the previous methods described in the data processing activity. In order to provide a feasible assessment, a modeling language need to model all the elements and artifacts in the education domain. The data model, described in Fig. 1, in its current state is not suitable for the assessment activity of the proposed framework, since it is detailed with insignificant, unclassified and nongeneric data. It is important to endorse the assessment activity with a modeling language that is formal, outcome-based and supported by a recognized tool. For that purpose, the Goal-oriented Requirement Language (GRL) has been selected. GRL is a model-driven language that is part of a standard modeling notation called URN and supported by an open source eclipse plugin called (jUCMNav, 2017). GRL allows to model conflict between goals and assists in making decisions that resolve conflicts. There are three main groups of concepts in GRL: Intentional elements, such as goal, soft goal, resource, task and belief; intentional relationships, such as contribution, correlation and dependency; and actors, such as User, System and Component. Table 1 describes the mapping between the data model Fig. 1  Courses are the physical objects selected for assessment and they are modeled as Resources. The Direct/Indirect Assessment Tools are represented in the GRL model as Key Performance Indicators (KPIs). A KPI contains an evaluation value that measures the accumulated students' results produced by the assessment tools. It is scaled from +100% (positive), +0% (neutral) up to -100% (negative) values. Constituencies are represented as Actors to represent the active objects that carry out actions to achieve the goal.
The data model in our proposed framework is not mapped to the entire GRL model elements. Some of model elements, such as belief and task does not have an equivalent in the data model. As a future work, extending our proposed framework by including the unmapped elements will be considered.

Evaluation Reporting
The evaluation reporting are generated using the model elements and performance measures of GRL goal model. Some of the reporting details are also extracted from the processed data. The reports may include charts and metrics that describe and visualize the current state and progress of an academic institution. It also supports the stakeholders and decision makers with data necessary for future forecasting. The assessment reporting can be presented in two formats defined as:  Executive dashboard: It is a strategic reporting dashboard tool that summarizes the institutional performance, metrics and real-time data in a form visual object. It also outline the big pictures of the institution against critical metrics, identify the opportunities of improvements and forecast for new investments  Performance reports: It is a report that handles the outcomes of an individual entity, such as activity, person, class or a program and performs a comparison between them. It can also be used to monitoring allocation and utilization of resources Framework Prototype Figure 4 shows the implementations prototype of our proposed framework.
The data collecting activity is supported by different artifacts. The word editors and viewers are used to build paper-based surveys and interviews. The SQL Server database contains a DBMS schema that defines various relational tables. The tables are used to store records that are related to the current students, graduated students. It may also contains tables for the institution structure, internal policies and regulations, HR and Finance. The academic portal provides multiple academic applications such as (Moodle, 2006). These applications can be used to form web-based surveys and forums. The data processing activity is supported by MS excel spreadsheet which provides many features such as data sorting and filtration, building formulas and equations and building pivot tables. Some of the artifacts at the data collection allows automatic migration of the data to the MS excel spreadsheet such as SQL server database and the academic portal. The rest of the data that comes from word editors and viewers are mapped manually.
The model-driven assessment activity supports a graphical editor for the GRL modeling language called jUCMNav. It is an eclipse plugin that provides several rich graphical formatting themes and coloring scheme. The scheme is used to measure and monitor the satisfaction degree of the model elements using different coloring shades for values scaled as: Unsatisfied (red), neutral (yellow) and satisfied (green). It also uses two kinds of evaluations: Quantitative measures (ranges from -100% to +100%) and qualitative measures (High, Medium and Low).
The model-driven assessment can be performed at different academic institution levels, such as course, curriculum, program and faculty. To simplify the assessment in this research, the goal model is generated at the course level to measure the achievement of students enrolled in one course or set of courses with respect to the SOs, PEOs and Program mission. Figure 5 describes an arbitrary goal model at the course level where a sample Course of the curriculum, represented as a resource. At the top level of the goal model, a Course contributes to a single PEO by 50%, which at the higher level contributes to the department Mission by 30%. Each course in the curriculum is assigned a number of SOs by matching between the course content and the common attributes of SOs (knowledge, solving problems, practical and communication skills, professional ethics, teamwork and leadership). Two student outcomes SO1 and SO2 contribute both to the Course by 25% respectively. There are also three assessment tools (KPIs): Assessment1, Assessment2 and Assessment3 are used for evaluating the student outcomes SO1 and SO2. They contribute by 50, 0, 75% and respectively. The owner of the goal model is the stakeholder. It represents the ownership and responsibility in the GRL model. Each model element in GRL has an Evaluation value that reflects its situation. Initially, the evaluation values of the KPIs (Assessment 1, Assessment 2 and Assessment 1) are defined based on the students' grades and then they will be reflected on the evaluation values of the top level model elements during the model analysis.
The evaluation reporting activity is the top level of the framework prototype, it supports an eclipse plugin called (BIRT, 2016). BIRT is a Business Intelligent (BI) tool that provides a data visualization technology and reporting. The data is embedded with a rich client platform and web application.

Case Study: ABET Accreditation of Computer Engineering Department
The case study has been developed at Faculty of Engineering in Al-Ahliyya Amman University (AAU) as a part of an on-going project. The objective is to provide a continuous improvement of the bachelor programs at the Faculty of Engineering and to qualify the programs for (ABET, 2019). The participated programs are: Computer Engineering, Civil Engineering, Communications and Electronics Engineering, Electrical Engineering and Medical Engineering. The framework has been applied on a sample of two courses: "Microprocessors" and "Intelligence Systems and Neural Networks" designated within the Computer Engineering program. The evaluation is performed on a period of two semesters: The fall and winter semesters of the year 2019.

Collection Data
The raw data of students' records and constituencies feedback are collected from different resources. Table 2 describes an example of the surveys used for the indirect assessments. The surveys are used to explore the satisfactions of different constituencies on the PEOs and (or) SOs of the program under evaluation. Alumni survey targets students who graduated since three to five years; Employer Survey targets the employers who hired graduated students from the program under evaluation; Field training surveys targets students who finished the training semester course; Student exit survey targets students who are in their last semester of study; Course assessment by student survey targets the enrolled students on every course in the semester and finally, Graduation project survey targets students who completed their Graduation Project course. The surveys are having structure, where the statements of the PEOs or SOs are listed in a table and a scale 1 to 5 is used to measure the satisfaction of the targeted constituency. Figure 6 describe a sample of filed training survey. We also add an additional column that measures the importance of the SOs based on scale 1 to 3. The importance measures are used for the continuous improvement and would indicate that more or less efforts need to be considered on the SOs during the semester.

Data Processing
The temporal analysis method is used to generate spreadsheets of the two course for the first and second semesters of 2019. A list of spreadsheet samples are described in the following figures. The spreadsheets described in Fig. 7 and 8 are extracted from the academic portal of AAU. Figure 7 describes the statements of the Mission, PEOs and SOs of the Computer Engineering Dept, while Fig. 8 represents the correlation between the PEOs and SOs (scaled from 1 to 100).
The grading sheet in Fig. 9 is extracted from the database and summarizes the grading of the courses in the Computer engineering curriculum for the first and second semesters in 2019. Multiple direct assessment tools have been applied with different marking scheme, such as homework, class work, project, quiz, midterm exam, final exam and others. For every assessment tool, there are two grades: The Max. Mark which represents the highest mark of the tool and Avg. Mark which represents the average mark of all enrolled students of the tool. The Avg. Mark is calculated as: where, k is number of enrolled students per course.  Our mission is to prepare distinguished graduates equipped with state-of-the-art skills that meet the need of marketplace locally and regionally and to encourage scientific research in the discipline.
Analyzing and providing solutions to challenging problems in professional career using computer engineering theory and practice.
Adapting, successfully, to technical changes in the chosen field and occupation.
Functioning independently, communicating effectively and working collaboratively with professionalism and ethical responsibility.
Providing technical leadership for business, profession and community.
An ability to identify, formulate and solve complex engineering problems by applying principles of engineering, science and mathematics An ability to apply engineering design to produce solutions that meet specified needs with consideration of public health, safety and welfare, as well as global, cultural, social, environmental and economic factors An ability to communicate effectively with a range of audiences An ability to recognize ethical and professional responsibilities in engineering situations and make informed judgments, which must consider the impact of engineering solutions in global, economic, environmental and societal contexts An ability to function effectively on a team whose members together provide leadership, create a collaborative and inclusive environment, establish goals, plan tasks and meet objectives An ability to develop and conduct appropriate experimentation, analyze and interpret data and use engineering judgment to draw conclusions An ability to acquire and apply new knowledge as needed, using appropriate learning strategies The spreadsheets in Fig. 10 describes the contribution weights of the SOs in the courses of the computer program (scaled from 0 to 100). These weights have been assigned using Round- Table Discussion and Consensus (RTD&C) approach (Akhigbe et al., 2014) mentioned before. The last spreadsheets in Fig. 11 are manually extracted from the survey documents filled by different constituencies and describe the average satisfaction measures (scaled from 1 to 5).

Model-Driven Assessment
The generated spreadsheets in the previous activity are mapped manually to the GRL goal model. It is worth to mention that all the data in the spreadsheets are scaled from 0 to 100 before the mapping. Figure 12 describes the goal model of the first semester 2019 generated by jUCMNav. The goal model of the second semester of 2019 is not shown in the paper.
The top model element is the mission of the computer engineering program. Three PEOs (PEO1, PEO2 and PEO4) contribute to the mission by 25%. The "0813405 Microprocessors" contributes to PEO1 by 90% and to PEO2 by 75%. While the "0815413 Intelligence systems and Neural Networks" contributes to PEO1 by 75%, PEO2 by 60% and PEO4 by 40%. Notice that there are not direct relations between the courses and the PEOs, however in this model we calculated them based on the contribution weight between the courses with SOs and the contribution weight between the SOs with PEOs as in Fig. 10 and 8. There are also three SOs (SO1, SO2 and SO7), where SO1 and SO2 contribute to "0813405 Microprocessors" course by 40 and 80% respectively, while SO2 and SO7 contribute to "0815413 Intelligence systems and Neural Networks" course by 80 and 60% respectively.
The evaluation of each assessment tool (KPI) is done through four value sets defined in the grading sheet, as in Fig. 9. These values are: (1) Evaluation value represents the Avg. Mark; (2) Target value represents the Max. Mark; (3) Threshold value is the minimum acceptable achievements of students under evaluation (not shown); and (4) Worst value is the most critical achievement of students under evaluation (not shown). There are five direct assessment tools, e.g. homework, midterm exam, quiz, final exam and project, represents the Key Performance Indicator (KPI). Each one of them contributes to multiple SOs that ranges from 30 to 75%. The Alumni survey is a sample of the indirect assessment tools. It contributes to all PEOs and SOs with contributions that ranges from 25 to 80%. Notice that the contribution weights between the assessment tools and the SOs are not shown in the data processing activity.

Evaluation Reporting
The development team uses Eclipse BIRT plugin to generate a suite of executive and performance reports for the ongoing project. A sample report generated for the temporal analysis of two courses in the first/second semesters 2019 is depicted in Fig. 13. The report presents the performance measures of different learning outcomes, objectives and assessment tools. It also visually compares between the two courses in the periods first/second semester of 2019.

Evaluation of the Proposed Model-driven Framework
The paper proposed a model-driven framework that supports four interoperable activities used to simplify implementing the evaluation process and reduce the complication and ambiguity of the analysis. The main target of the framework is to generate executive and performance reports that outline the institutional status, progress, metrics and real-time data in a form visual object. The reports supports the stakeholders and decision makers with data necessary for continuous improvement and future forecasting.
The related work mentioned before proposed assessments are paper-based, such as rubric, where documents are used in the evaluation process of learning outcomes. We developed an integrated hierarchical structure framework that supports a model-driven assessment using GRL goal modeling in evaluating the learning outcomes. This will improve the evaluation process and assessing the learning outcomes and objectives in formalized manner and reduce the complication and ambiguity of the analysis. The framework also supports generating executive and performance reports that help decision maker in monitoring the institution progress and data necessary for future forecasting. Figure 13 describes a sample of performance report for the Computer Engineering Dept. at AAU. The report presents the performance measures in the first and second semester 2019. It also allows to compare between the performance measures necessary for continuous improvements.
The report shows that there is a slight reduction in the measures of the mission, POE1, POE2 and SO2. However, there is an improvement in the measures of the SO1 and SO2. As for assessment tools, we found that the contribution of quizzes and project has increased caused by the students' achievements. Based on the big picture of the performance reports, a list of recommendations has been released as part of the continuous improvements for different constituencies.
For Current Students: 1. Focusing more on practical aspects and problem solving skills through the core and elective courses 1. There is a large demand on quality-related functions such as quality assurance, validation and HW/SW testing 2. Students should be exposed to relevant topics to prepare them for such functions 3. More emphasis on the business/economic side of engineering 4. Encouraging the innovative personality within the graduated engineers For Employers: 1. Improving the communication skills of the students/engineers 2. Improving the problem solving skills and initiative among students 3. Focusing more on the leadership and management aspects Several challenges were addressed initially due to large number of participated constituencies and lack of quality former performance measures. During the evaluation process, teams from different disciplines are required to meet periodically to discuss various topics related to the framework activities, such a kind of direct/indirect assessment tools used, a type of analysis method, GRL modeling structure, modeling elements and relationship between them and metric values of the model elements. This may increase the chance of human error and increase the period of becoming familiar with the framework activity.
Challenges were also faced in handling and processing high-volume and complex raw data. The raw data need to be categorized based on its importance to the evaluation process and then migrated into different spreadsheets. The migration can be automatic for the structured data, such as database or manual extracting for paper-based documents. The spreadsheets are built to meet required analysis methods, i.e., cohort, spatial, temporal and multi-level management. Other challenge is related to how much the measure of the model elements are accurate. It is obvious that the validity of GRL models depends on the accuracy of the model element measures. Though, we found based on our practice that the accuracy of GRL modeling results deviate towards the improvement as the time proceeds and the participated constituencies are familiar with approach.
However, despite the initial challenges mentioned before, we are able to overcome these difficulties by managing the framework properly. Also, when constituencies become familiar with the framework and gaining skills and practices of using the applications of the prototype.
The framework also has several limitations caused by the nature of handled data. First, the framework cannot be used in evaluating behaviors and procedures that may affect the institution mission, PEOs and SOs, such as the Standard Operating Procedures (SOPs). This limitation will be considered in the future work of the project. The framework also does not provide a full automation of data collection, migration, processing and mapping to GRL models. Different format of collected data are migrated, processed and mapped manually. The framework also has a sort of restrictions in the mapping between the data model and the GRL model caused by the limited number of model elements of the GRL.

Conclusion and Future Work
Educational institutions are adopting different learning assessment methods and techniques to evaluate and improve the student learning outcomes. The evaluation of the learning outcomes is an ongoing enhancement process conducted at the educational institutions and begins when student enters the college until the time of graduation. The evaluation process can be complex and challenging. This is mainly caused by the nature of the raw data used in assessment. The data is usually unstructured, complex, text-heavy and collected in high volumes. It may also be extracted from heterogeneous platforms and require privileged accessibility when it comes from different management-levels of the institution. Also, different paper-based assessment tools, such as Rubric, are used in the evaluation process. Using paper-based assessment in complex evaluation process may cause error prone, confusion in analyzing the learning outcomes and subject to different interpretations of the assessment by academic constituencies. We propose a model-driven framework for evaluation process of the learning outcomes. The framework consists of four activities: Data collection, data processing, model-driven assessment and evaluation reporting. The data collection and data processing activities are used to extract complex data into a useful information for assessment. The model-driven assessment activity is used to generate and analyze goal models of the learning outcomes augmented with quantitative indicators. The generated goal models improves the assessment process, evaluate the learning components in a formal way and allows the assessment at different level of academic institutions. Finally, evaluation reports are generated to summarize the institutional status, metrics and real-time data in a form visual object. The reports include charts and tables that communicate how well the students' achievement and feedback is contributing to learning outcomes and objectives.
As a future work, additional case studies are going to be implemented using the proposed framework based on different data processing analysis methods, such as cohort, spatial and multi-level management data analysis.