A Framework for Computerized Progress Feedback in Tertiary Education

Corresponding Author: Annette van der Merwe Department of Computer Science and Information Systems, North-West University, Potchefstroom, South Africa. Email: VanDerMerwe@nwu.ac.za Abstract: The dawn of democracy in South Africa in 1994 has resulted in challenges in many fields, especially in education. University enrolment rates are high and the output quality of school learners in general is delicate which threatens the graduate quality of tertiary institutions. Contrary to rising student enrolments, the attrition rates have not improved. One contributing factor is the lack of comprehensive insight into students’ academic performance throughout the course of a program. New systems are being developed and implemented in both online and on-campus programs and can be utilized to improve student performance feedback. Prior studies served to identify the ideal elements, models and concepts necessary to develop a framework for academic progress feedback in the tertiary education environment. In this paper, a model-based academic progress feedback framework is proposed to improve learning. The framework was implemented in a case study at a South African university and evaluated using a survey. The framework was readily accepted by the end-users and conformed to the features and requirements identified in literature.


Introduction
Since the first democratic election in South Africa in 1994, changes have occurred in many research fields, especially in education (Suransky and Van der Merwe, 2016). Research on education in South Africa remains topical and range from studies in ethnicity (Jawitz, 2012), gender factors (White et al., 2011) and free education (Nordstrum, 2012), among others. The increase in university enrolment rates as opposed to the generally weak output quality of Grade 12 learners, which in turn lowers the standard of graduates delivered by tertiary institutions, is and remains a challenge (Modisaotsile, 2012;Shay, 2017).
Successful learning depends on efficient teaching, technological advances, student participation and other factors. The main role players in a tertiary teachinglearning environment are the lecturers and the students. The purpose of both is to accomplish successful learning and help improve student achievement. One of the most important factors to stimulate effective learning, is feedback (Hattie, 2009).
Feedback in learning comprises information that a lecturer or teacher provides a learner with, relating to accomplishment or content understanding (Hattie and Timperley, 2007). Although academic performance feedback and its effect on learning have been extensively researched in recent years (Belcadhi, 2016;Semley et al., 2016;Stegemann and Malan, 2016), students' learning methods have started to change. Specifically, computer and internet use in learning are increasingly being adopted (Uğur and Turan, 2018). Feedback on learning and academic progress, therefore, also needs to adapt to cater for the trends that are being followed by students.
Learning analytics entails collection and analysis of teaching and learning data, purposing to deliver positive and negative feedback to students and lecturers (Fiaidhi, 2014). Decision Support Systems (DSSs) are commonly used to aid in decision-making, especially in rapidly changing environments and present the ideal tools with which some learning analytics functions can be implemented (Fadiya, 2017).
The inspiration for this study was founded in the observation that the quality of student learning on tertiary level in South Africa is weakening, although the resulting enrolment numbers are escalating annually. This paper represents the culmination of a series of studies in which: Mathematical models were utilized, validated and evaluated as a learning analytics tool to rank students according to academic performance (Van der Merwe et al., 2016a;2016b); the tool was implemented into an academic feedback system for timely progress reports (Van der Merwe et al., 2017b); the feedback system was optimized and expanded to include a dynamic academic improvement calculator for individual students (Van der Merwe et al., 2018a); and the optimized feedback system was implemented in a specialized learning management system for validation and evaluation (Van der Merwe et al., 2018b). The aim of this study is to develop a modelbased academic progress feedback framework for wide scale implementation to improve student retention rates at universities. This will be achieved by reaching the following secondary objectives: • Combining the knowledge gained from previous studies with that of related research in a short literature study; • Composing and illustrating the academic progress feedback framework; • Implementing the framework in a tertiary education environment; and • Evaluating the framework according to the constructs identified in previous research and literature.
The research method is outlined in the next section followed by a literature study on educational feedback, learning analytics and the use of a DSS for learning analytics functions. Thereafter, the feedback scenario and evaluation measures according to which the proposed framework was assessed, is discussed. The new feedback framework is then presented, after which validation and formal evaluation of the framework is discussed. Research contributions are stated and concluding remarks made in the last section.

Method
This study was performed within the positivistic paradigm for Information Systems as described by Siponen and Tsohou (2018). A combination of qualitative and quantitative data collection methods was used. More specifically, the proposed framework was developed according to an action design research method which consists of four stages (Wing et al., 2017).
In the problem formulation stage, the collective knowledge gained from previous studies and literature was combined to construct an academic progress feedback scenario and compose the evaluation criteria for a feedback framework. The building, intervention and evaluation stage included establishing the feedback framework requirements and constructing the metadata model components. This was followed by the implementation of the framework in a case study at a South African university and evaluation according to the factors identified in the first stage.
The instrument of data collection was a survey in the form of a structured questionnaire. A convenience sampling procedure was used due to the students participating voluntarily and anonymously. The reflection and learning stage is covered in the discussion section and formalization is completed by means of construction of this paper.

Related Work
Numerous research projects have been completed on what feedback entails and what issues it should address to be effective (Hattie and Timperley, 2007;Wiggins, 2012). The Oxford dictionary defines feedback as "information about reactions to a product, a person's performance of a task, etc. which is used as a basis for improvement" (English Oxford Living Dictionaries n.d. Feedback, 2018). Wiggins (2012: 11) defines feedback as "information about how we are doing in our efforts to reach a goal." The specific purpose of feedback is clearly stipulated as something to be used for improvement towards reaching some objective.
In the educational environment, assessment feedback is a widely researched field (Belcadhi, 2016;Evans, 2013;Krause et al., 2009;Miller, 2009). The term broadly encompasses assessment design in a learning environment so that feedback relating to the posed questions is created at some stage during assessment (Evans, 2013). Some believe that feedback is more valuable to individuals than to groups (Krause et al., 2009). The effect of feedback is determined by its design and by the way in which it is received and acted upon. Miller (2009) introduced a formative computerbased assessment framework aimed at improving selfregulated learning and used feedback as an instrument to support student learning. Although the utilitarian value of assessment feedback was investigated, one of the findings was that the students thought feedback should occur soon after an activity rather than being delayed. The collation of feedback should therefore be done in as short a time as possible to benefit students.
Whatever the objectives are when studying feedback, researchers agree that it is an important factor in improving learning. Wiggins (2012) argues that endless feedback is more important to learners than endless teaching. Academic progress feedback is not necessarily subject related but rather provides information on how a student is performing or progressing in the aim to pass a module. The process of generating such feedback can be time consuming, especially with large student numbers (Krause et al., 2009). Modern analytics techniques offer attractive methods for processing and analyzing large data volumes thereby converting them into insight (Daniel, 2015). The demand for improved productivity and achievement is progressively being addressed through incorporation of different forms of analytics into educational systems. These include institutional, information technology, academic and learning analytics, among others.
Learning analytics is concerned with the "measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimizing learning" (Daniel, 2015: 913). It can be implemented to generate feedback on learners' academic progress thereby promoting evidence-based decision-making. Higher education institutions are increasingly implementing big data analytics tools to assist them in decisionmaking (Rubel and Jones, 2016). Ruipérez-Valiente et al. (2016) found that although Learning Management Systems (LMSs) collect different forms of student data, not many provide appropriate visualizations about the learning process. The purpose of learning analytics systems can vary but to improve learning, it should assist in the decision-making process and also help with detecting problem-cases early on. It was found that using learning analytics should be extended to not only augment the teacher's information but also provide students with selfawareness tools (Ruipérez-Valiente et al., 2016).
The value of effective feedback depends on numerous variables but in particular on the timely availability of data as well as the uniqueness of the information that can be provided (Tempelaar et al., 2015). Model-driven DSSs use optimization, algebraic, financial, analytic, or simulation techniques to provide decision support (Power and Sharda, 2007). Quantitative models implemented in a DSS that performs learning analytics functions, present a unique solution for timeous data processing (Power and Sharda, 2007). The implementation of learning analytics in a modeldriven DSS will offer advantages to lecturers and students alike, irrespective of their technical capabilities. "Models in a model-driven DSS should provide a simplified representation of a situation that is understandable to a decision maker" (Power andSharda, 2007: 1045). The learning analytics functionality in the progress feedback framework presented in this paper, is therefore provided by a model-driven DSS which uses quantitative mathematical models to manipulate the data.
To evaluate the proposed progress feedback framework, the properties of effective feedback need to be investigated. Researchers generally agree on the conditions under which assessment feedback has a positive effect on learning (Belcadhi, 2016;Evans, 2013;Krause et al., 2009;Rubel and Jones, 2016;Wiggins, 2012). Academic progress feedback differs from assessment feedback in that it is based on overall student performance in a module, rather than on individual assignments. The proposed framework was developed according to a qualitative technique proposed by Jabareen (2009) and is model-based because it consists of constructs and variabledependent models. The evaluation criteria for the proposed framework is a compilation of the conditions established in literature relating to progress feedback, privacy concerns in learning analytics and the behavioral and technical issues concerning a model-driven DSS.

Progress Feedback Scenario
In order to identify and integrate the elements relevant to the proposed framework, the academic progress feedback scenario is described and analyzed in the next section, followed by the evaluation criteria for the framework thereafter.

Scenario Layout and Analysis
A lecturer is responsible for facilitating module contact sessions and determining how to grade and pass students. On-campus students are expected to prepare for and attend contact sessions, study subject material and complete formative assessments like attendance, informal tuition or formally scheduled tests. Assessment grades are entered into a spreadsheet in which students are typically identified by their university numbers. A collection of grades as progress feedback, lacks essential information that students need to improve their learning. Feedback has a positive effect on learning when it reports backwards (provides each student with up-to-date progress reports), upwards (gives the student a goal to work towards) and forward (lists distinct actions for a student to perform towards reaching the goal) (Hattie and Timperley, 2007). Therefore, in the proposed framework a DSS performs the data analysis and generates a personalized progress profile for each student, with these three forms of feedback.
A detailed description of the mathematical models and the related algorithm implemented in the DSS, can be found in Van der Merwe et al. (2018a). The progress profile of a student is generated using all accumulated grades at any current point in time in the semester. The DSS requires certain module specific information from a lecturer to generate an improvement plan for each student. This information includes the type of assessments (i.e., class assignments, tuition tests, etc.,) as well as the number of assessments in each category. The calculated weighted average participation mark (a student's level of participation in academic activities) is used in conjunction with a layout of the remaining planned assessments for a semester to determine if a student can improve by a specified amount, like 5% (provided by the user). A decision-tree structure is created by the DSS to represent all the possible scenarios that a student can follow for participation in the remaining assessments. For each one of those scenarios, the algorithm calculates a required score per assessment that will result in an increase in the current weighted average participation mark, as provided.
Learning analytics techniques implemented on student data sets, present privacy issues that need attention (Rubel and Jones, 2016). An academic ranking that categorizes students according to their progress relative to the rest of the class, partially addressed this issue (Van der Merwe et al., 2016b). The method allowed students to compare their progress with that of their peers without actually knowing the grades of the rest of the class.
Feedback can be provided to users by means of any existing platform used for communication of academic information, like an LMS. A student accordingly has several options to consider when proceeding with the semester. A typical student progress profile is shown in Table 1.
Apart from the information in Table 1, a lecturer has access to the assessment class averages which can be used to determine fixed weights towards reaching a specific participation mark class average (Table 2).
If, for example, a lecturer wishes to maintain a class average of at least 70% the DSS can calculate the weights required for each assessment type, based on the students' current marks.
Participation profiles/semester marks are used as the measure by which students are allowed admission to the final summative assessment/examination. The final module mark is calculated according to a mathematical equation established by the lecturer, by considering the semester mark and that of the final examination.
The outlined scenario presents some challenges. Research indicates that the frequency of academic progress updates is generally unsatisfactory (Carless, 2006;Gielen et al., 2010). Students fail to obtain their assignments or grades timeously (Carless, 2006) and remain oblivious to impending failure (Du Toit, 2015). Also, lecturers often have trouble identifying at-risk students when trying to process large data sets in a limited amount of time (Van der Merwe et al., 2016b). Feedback should ideally be provided to students and lecturers in time for them to still be able to affect change.
In any teaching-learning environment, the persons responsible have to manage various challenges to facilitate effective feedback. Specific issues and their framework references were identified in this section and will be used for evaluation of the proposed framework. These concerns were translated into framework requirements and are discussed next.

Requirements of the Proposed Feedback Framework
A framework is a grid with interlinked elements consisting of concepts and/or models that collectively provide comprehensive insight into a phenomenon (Jabareen, 2009). Conceptual analysis of the discussed scenario resulted in the identification of the following components: the students, data, a lecturer, a DSS and an LMS for providing feedback. These elements were combined to portray the academic progress feedback scenario discussed previously.
Feedback in general should exhibit seven key attributes to be effective (Wiggins, 2012). Those relevant to academic progress feedback are timeliness, consistency, being goal-referenced, generic (independent of subject content), personalized and actionable.
Utilization of a DSS facilitates compliance to all of these requirements. Incorporation of a DSS to perform the necessary analytical actions on the data set decreases the amount of required processing time considerably. This allows a lecturer to provide progress feedback consistently and as soon as the assignments are graded. Once the captured data have been processed, feedback should report backwards, upwards and forward to students and lecturers alike. The resulting feedback is actionable because both students and lecturers can subsequently plan and act to affect improvement.
A model-driven DSS allows a non-expert user to set up or manipulate the parameters required to solve the models (Power and Sharda, 2007). The models are dependent on the knowledge and experience of the perceiver and will therefore be evaluated for conceptual ease, basic operation and model support (Argent et al., 2006).
In summary, feedback provided to the user should: • Be provided timely and consistently; • Protect student privacy while providing an academic ranking; and • Report backwards, upwards and forward.
Certain semantic models need to be preserved when setting up the concepts and models necessary to provide insight into the described feedback phenomenon. These models and how they are interlinked to epitomize the feedback environment, follow.

Development of the Academic Progress Feedback Framework
The scenario outlined previously can be categorized into three interlinked phases namely "perform", "analyze" and "report". These phases represent the semantic models that need to be maintained within a feedback framework in order to adhere to the requirements as set forth in the previous section. A description of the semantic models and their attributes and how they were integrated into a new feedback framework, follow.

Models Description
The three phases identified served as the blueprint for developing the framework for personalized academic progress feedback. The "perform" phase involves face-toface contact between the lecturer and students. An individualized user profile is created for each student which is a structure for modelling and storing information relating to a student's current academic progress and characterizes the participation and performance level of a student. The user model therefore portrays a student in terms of a participation level and ranking relative to the rest of the class. This data is categorized as personal, performance and profile information.
The captured data is used in the "analyze" phase to generate progress feedback and convert it into an appropriate format. A metadata model describes the entities involved in data generation and the processes of producing, capturing and formatting the data into the structure required for feedback. The metadata model is implemented by means of a DSS because it interacts with a decision maker and applies various different methods to provide information that will support decision-making (Chiu and Huang, 2016).
The model-driven DSS implemented as part of the proposed framework, maintains the metadata model so as to generate feedback and format the information into the required form. The DSS implements the mathematical model in Van der Merwe et al. (2018a) to create a progress profile for each student consisting of: • A set of all the accumulated marks; • A dynamically weighted participation mark average calculated from the current marks; and • An overall student ranking created from the student participation mark averages.
After completion and grading of each new assessment, all of the progress profiles are re-calculated and updated information is provided to the students. As the semester progresses, students have the ability to monitor their academic performance.
In order to provide all the relevant users with personalized (for students) and generic (for a lecturer) feedback in the "report" phase, the strategy and resource(s) need to be appropriate to the given context. These must be maintained in the feedback model. The general ontology between these models for a number of n assessments and m students, is illustrated in Fig. 1.
The feedback strategy implemented in the DSS, provides information granularity by presenting progress profiles in terms of a layout of grades that a student achieved for each assessment. LMSs are widely used for communication between a lecturer and students in tertiary education environments but generally lack specific integration of academic feedback to students (Stantchev et al., 2014). A Specialized Learning Management System (SLMS) is ideal for providing feedback to students as it allows a lecturer to use it for data capturing and access to progress profile information (Van der Merwe et al., 2018b). This will enable a lecturer to identify at-risk students and implement timely interventions to assist them. The use of an SLMS also contributes to the protection of student privacy by requiring authentication before gaining access to individualized student feedback. In this section, the models and their relationships were discussed for the purpose of recognizing the elements needed for the proposed feedback framework. These elements and how they combine to form the new feedback framework, follow in the next section.

The Academic Feedback Framework
The elements identified for the feedback framework include concepts and models as part of the DSS. Each of these elements provides some contribution towards the feedback framework in terms of ontological (the nature of reality or actual existence and actions), epistemological (the way things work in an assumed reality) and methodological (the process of constructing and assessing information about the performance status of students) assumptions (Jabareen, 2009). The elements required for efficient depiction of the three phases identified in the previous section, are shown in Table 3.
Frequent academic feedback can improve a student's academic motivation as it reveals the effect of student participation in academic activities. A motivated student exhibits a change in academic conduct. A lecturer also expects to see certain results from the teaching methods employed during contact sessions. Feedback can affirm that these methods are successful and if not, prompt the lecturer to adapt. The elements were synthesized into a theoretical modelbased conceptual framework for ranking feedback and is presented graphically in Fig. 2.  The framework provides comprehensive insight into the feedback phenomenon in the described teachinglearning environment. It contains the required structures, effectively addresses the issues that were identified and maintains the relevant semantic models. In order to establish whether the framework comprehensively portrays the academic feedback phenomenon, it was implemented at a South African university and evaluated according to the requirements identified in the previous section. Framework validation and evaluation is discussed next.

Evaluation of the Proposed Feedback Framework
The proposed framework is model-based and consists of several interlinked concepts. Validation of the evolving theoretical framework was done by means of peer reviewed presentations at academic conferences in management science (Van der Merwe et al. The qualitative measures for evaluating the DSS include conceptual ease, basic operation and model support. The DSS consists of mathematical models that were designed so that student or lecturer-users can have access to its modelling capabilities without having expert knowledge of the mathematics embedded within. Although the conditions for module completion can vary, the proposed framework utilizes the data generated by students as variables rather than set parameters, according to which each student's progress is determined. This means that the progress feedback framework is a generic solution, can be set up according to specific lecturer requirements and be implemented in any module, regardless of the content. The DSS was created as a standalone computerized system which means further modelling support will only be required for exceptions or special cases. Formal evaluation of the framework was done by implementation in a third year Computer Science module. The feedback platform used was an existing SLMS used at the university. Students were asked to participate in the three phases along with a lecturer and evaluate the feedback by completing a survey. The constructs employed in the survey (Table 4) were inferred from the criteria set in the previous section.

REPORT
The students were invited to show their level of agreement with the posed constructs on a Likert scale ranging from one (totally disagree) to seven (totally agree). In a module with 129 enrolled students, 97 (75%) completed the survey. A level of satisfaction was calculated for each factor by determining the ratio of students who responded with values five through seven. The level of satisfaction for the frequency and timeliness of the provided feedback (TC), as well as the effectiveness of peer Ranking (ER), was 92%.
The students' overall satisfaction with the content of the feedback given (BFU) was 94%, indicating that they felt that the information supplied will help them to improve their academic progress during the semester. Anecdotal evidence from informal class discussions indicated that although their academic rankings provided students with some motivation to improve, they wanted to have access to the detailed grade sets of their peers for comparative purposes. Furthermore, respondents indicated a level of 95% satisfaction with the confidential use of their academic data. Figure 3 shows the averages of all the students' responses, for all the evaluated constructs.
This section started with an outline on the validation of the proposed feedback framework by means of presentations of the evolving concepts at academic conferences, followed by a brief reference to the evaluation of the mathematical models implemented in the DSS in terms of qualitative measures established by Argent et al. (2006). The section concluded in a discussion on the formal evaluation of the feedback framework as implemented in a tertiary education setting. A layout of specific and more general contributions of this study to the field of educational feedback, follows.

Discussion
One of the major success factors towards improvement in the learning process in tertiary education is effective academic feedback. Research has also shown that in order to be considered a leader in teaching, lecturers and institutions need to efficiently incorporate technology in their teaching regimes (Newman and Scurry, 2015). An academic feedback framework was proposed that provides insight into the changing tertiary education environment. This study offers goal specific contributions and some more general contributions.

Specific Contributions
The proposed framework provides timely feedback in a dynamic teaching-learning environment. The feedback contains up-to-date progress reports, individual goals to reach to affect improvement as well as distinct actions to perform towards reaching the set goals. A progress report consists of a compilation of the grades a student achieved during a semester, an updated minimum participation mark and a ranking position for that student in relation to the rest of class. The improvement goals (feed upwards) contribute towards improved learning in that students can change their approaches towards learning by following the steps suggested (feed forward). The structure of these performance reports is in contrast to the traditional manner of providing students with only their latest assessment grades and contributes towards improving student motivation.
Implementation of the proposed framework improves module management by lecturers. Academic feedback is provided timely, leaving lecturers with more time to spend on expanding their teaching regimes and providing facilitation for hard-to-master subject content. Lecturers have access to student progress information which can help them identify at-risk students and implement interventions early in a semester.
Although the framework was evaluated in a contact program, it has the potential to improve the management of online courses. Such courses are generally operated by means of LMSs on the Internet. An SLMS was demonstrated in this study to be the ideal deployment platform of the proposed framework. The feedback framework successfully integrates existing concepts with new approaches to efficiently address the dynamic nature of the evolving teaching-learning environment.

General Contributions
The proposed academic feedback framework successfully integrates concepts and models to describe the intricacies of educational feedback. For this purpose, a combination of the qualitative methods proposed by Jabareen (2009) and Argent et al. (2006) were applied. These studies were also used to set up appropriate evaluation criteria for the developed framework.
The proposed framework effectively supports the goal of improving student achievement in an evolving teaching-learning environment and assisting lecturers with early identification of at-risk students. The provided feedback can also be used to identify subject content that students have not fully mastered.
Implementation of the framework will contribute on an institutional level to the quality and accuracy of learning statistics. The framework also contributes to the field of learning analytics by using a DSS to perform the data processing. The DSS can be implemented as an educational tool so that users with no mathematical modelling knowledge can operate it, ensuring dynamic, timely and up-to-date academic performance feedback in many modules.
The proposed framework will help to improve communication between institutions and students, so too in online programs. By providing feedback that is relevant to their learning progress, an institution can positively influence the students' motivation towards their studies. Implementation of the proposed framework in South African universities will therefore help to improve student retention which in turn can improve the overall quality of graduates.

Conclusion
The aim of this study was to develop a modelbased academic progress feedback framework for improving the learning standard of tertiary education in South Africa, in general. This was done by reaching some secondary objectives. Recent research relating to feedback, learning analytics and DSSs were investigated in a literature study to form a greater understanding of the environment and how it is changing. The concepts and models relevant to this environment were identified, categorized and integrated to compose and illustrate the academic progress feedback framework. Academic presentations served as validation of the evolving theory, the framework was implemented in a case study and evaluated according to the constructs identified in literature.
As the DSS analyses academic data, the need for further research into improving student data privacy, exists. Therefore, the platform for mass deployment of the framework for many modules within an institution and the access measures used to authenticate its users, must be reliable and conform to the requirements for the protection of personal privacy. Research and development regarding privacy issues and the use of LMSs for the proposed framework as well as investigation into the success of mass framework deployment at a South African university, serves as future work.