Achievements and Usage of Learning Materials in Computer Science Hybrid Courses

Corresponding Author: Andrija Bernik, University North, Department of Multimedia, Ul. 104. Brigade, 42000 Varaždin, Croatia Email: abernik@unin.hr Abstract: The term gamification is used to describe the implementation of computer game elements into non-game-based systems. We have expended our previous research and in this article we’ve introduced new concept model for comprehensive gamified approach based on literature findings. Next, we empirically tested two hypotheses regarding the learning achievement in specific topics of online university courses “3D Modeling” and “Programming II”. These hypotheses were: H1-an online course designed with the use of computer game elements enables better learning achievement in comparison to traditional non-gamified e-learning and H2an online course designed with the use of computer game elements positively influences the frequency of use of the learning material. Both hypotheses were confirmed, based on which final conclusions were drawn regarding the use of gamification in computer science hybrid courses.


Introduction
Before defining "gamification," it should be noted that it designates the introduction of some elements of computer games into the non-play area. This term was first coined by Nick Pelling in 2002 (Schönen, 2014;Jakubowski, 2014). Weiser et al. (2015) was stated that until 2010 gamification had not been extensively used or thoroughly investigated, mainly due to the slow application of information and communication technology for this purpose. Deterding et al. (2011) was emphasized that gamification as a process of introducing computer game elements into a non-play area represents an extremely fast growing trend in the educational field (Deterding et al., 2011;2012;Schell, 2014). Gamification can be defined from different points of view but, according to (Lombriser, 2015), it denotes "using the mechanics of computer games and the user interface for the purpose of digital involvement and motivating users to achieve their goals." The main elements of this definition are (Lombriser, 2015): Computer games mechanics; User interface (design); Digital engagement; Motivation; Goal achievement. Saeter and Valle (2013) established that "gamification is a phenomenon that uses computer game mechanics, aesthetics and logic to incorporate and motivate users and focus them on learning and problem solving." Souza-Concilio and Pacheco (2013) was claimed that the implementation of computer game elements is visible in various areas including education, health and fitness, task management, environmental sustainability, science, user generated content etc.

Related Work
Both in literature and in educational practice the principles that are similar to gamification are collectively referred to as "serious games," representing the application of computer games for learning of specific skills rather than for entertainment or enjoyment. Most recently, serious games have come to denote education and training regarding work-related skills and tasks that are solved by using a computer game. Therefore, a serious game is a computer game whose primary purpose is not entertainment but learning (Hakulinen, 2015). The relationship between gamification and serious games (Schönen, 2014) is shown in Fig. 1.
A game is defined by rules, competition and desire to achieve a specific goal. According to (Versteeg, 2013) and (Shabihi et al., 2016), a game can incorporate three basic elements: points, badges and a list of achievements. These three elements are known as the Point, Badge, Leaderboard system (PBL system), where ■■ points are attained by some measurable activity, badges visualize a specific achievement and represent an acknowledgment to the user, while a list of achievements serves to compare participants (Versteeg, 2013;Mekler et al., 2017;Lombriser, 2015;Werbach and Hunter, 2012). Lombriser (2015) extends the PBL system and states that, in order for something to be played, the following six components must be present: (1) rules; (2) variables and a clear outcome of the game; (3) success that is dependent the leaderboard; (4) challenge-i.e., varying complexity/difficulty depending on the progress; (5) the attribute clearly identifiable parts of the game; (6) reality. According to (Hamari et al., 2014) a similar list of elements characterizes a gamified approach, including: points, top-list of achievements, badges, levels, visual progress through the system, user success, motivating story, clear goals, feedback, awards system, avatars, adventures and dynamic development of the user profile. Finally, (Saeter and Valle, 2013) suggest that each game should attract its players in a voluntary manner and, once they are in the system, keep them engaged for as long as possible.
In their highly-cited paper (Deterding et al., 2011) describe gamification as a combination of the following three concepts/attributes: (1) mechanics, (2) dynamics and (3) aesthetics, which are aided by graphics elements from computer games. Mechanics (M) represents system rules, Dynamics (D) is associated with the interaction and behavior of the system, while Aesthetics (A) encompasses enjoyment and fun elements of the system. This categorization has been confirmed by other authors such as Zichermann and Cunningham (2011). In a similar approach, computer games are conceived as the combination of four equally important components, namely, technology, aesthetics, mechanics and story (Schell, 2014). A new model of the gamification approach, based on DMA, according to guidelines in (Werbach and Hunter, 2012) and (Deterding et al., 2011;2012) is presented in Fig. 2.
The most common gamification elements that can be used in business and education systems include reward symbols, status, personal and points. It must be noted that an adequate use of computer game mechanics for educational purposes presupposes familiarity with the technological capabilities of the educational platform (for instance, a specific learning management system) as well as knowledge of how the available mechanics work and how a specific technological solution can be used to influence learners' thinking processes and motivation. According to (Mayo, 2007), the mechanics of computer games in educational systems commonly provide features such as: a) Experiential learning. Engagement in activities according to different scenarios that require personal decision making b) Inquiry-based learning. The use of free-form exploration and experimentation c) Self-efficacy. Encouraging users to continue to use a system as long as possible by means of incentives in form of points, levels or "magic swords" d) Clear and precisely described goals. Learning is more efficient if the reasons for performing activities are easily understood and relevant e) Co-operation. Teamwork focused on joint problem solving or task execution may result in increased achievement in comparison to individual or competitive learning f) Continuous feedback. Tracking students' achievement, providing frequent and timely feedback and subsequent tailoring or adjustment of further learning experiences The use of gamification in teaching programming or computer science courses has recently been a topic of several empirical studies (see : Fresno et al., 2017;Arawjo et al., 2017;Fotaris et al., 2016;Ibáñez et al., 2014). However, several shortcomings of these studies can be identified: (1) using a limited number of gamification elements (e.g. badges and leaderboards); (2) no control group to compare the effects of gamification; (3) the research not performed in two phases -i.e. a pilot study and main study. Therefore, we find our approach, as presented in Table 1, to be a useful contribution to the body of research on the use of gamification in programming and computer science education reported in literature. It must be noted that most of the previous studies have established a positive effect of gamification on students' learning and motivation although these may also depend on the learning context as well as values and attitudes of students.

Research Plan
The main focus of our research was to test the effectiveness of introducing gamification into an elearning component (e-module) of a hybrid university course on programming. The criteria for the effectiveness of gamification were (a) higher test scores and (b) more frequent access to online study materials. The following two hypotheses were formulated in relation to the previously mentioned research problem: H1: An e-learning course that is designed with the use of elements of computer games (in the experimental group) will result in greater student achievement regarding test results in comparison to the use of traditional non-gamified e-learning course (in the control group). H2: The use of an e-learning course with elements of computer games (in the experimental group) will have a positive effect on the frequency of use of the teaching material in this course in comparison to an e-learning course with equal teaching material but without the use of elements of computer games (in the control group).
Our research was conducted in two phases: a pilot study with 55 subjects and a main study with 201 subjects. The research methodology is explained in more detail in the continuation of the paper.

Research Methodology
In the pilot study two e-learning modules (gamified and non-gamified) were created for both the experimental and control group of students with equal theoretical content on the specific topic "Lighting and Rendering" of the university course "3D Modeling". In the main study two new e-learning modules (also gamifies and non-gamified) with equal theoretical content on the topic "Heap and Stack" of the university course "Programming II" were created for the experimental and control group, respectively as it is described in Bernik et al. (2017). The Moodle Learning Management System (LMS) was used in both studies and both types of e-modules-i.e., gamified and nongamified. In Table 1 the gamification elements which were implemented into the e-learning modules that were used for the experimental groups of subjects are compared with the elements of the e-learning modules that were designed for the control groups.    Table 1, the experimental group in the main study used an e-learning module on the topic "Heap and Stack" with a total of 24 gamification related elements, while the control group in the same study used an e-learning module on the same topic, but with only 3 gamification elements (avatar and personal information, asynchronous communication forum, nonlinear use of teaching materials). In the pilot study the number of gamification elements for the e-learning module "Lighting and Rendering" was slightly lesser for the experimental group of students but it still included a total of 17 gamification elements.

Participants and Groups
The subjects (participants) in the pilot study were 55 sophomore students who attended the elective university course "3D Modeling" at a recently founded Central European university. This convenience sample consisted of male (56%) and female (44%) students. Also, the sample encompassed full-time (33%) and part-time (67%) students. The subjects were divided in 4 study groups of comparable size. Two study groups were treated as the experimental group and the other two were treated as the control group in the empirical part of our pilot study. The initial pre-test showed no statistically significant difference between the experimental and control group of subjects regarding their previous knowledge of multimedia technology. The subjects (participants) in the main research were students enrolled in the university course "Programming II" at a very large and well-established university in Central Europe. There were a total of 201 subjects who voluntarily participated in this study. The sample consisted of male (78%) and female (22%) students aged 20 years on average, almost all of whom were full-time students. The participants were divided in 14 groups of 14-15 students each for attending seminars in computer laboratories. Seven of those study groups were treated as the experimental group, while the other seven constituted the control group. Similar to the findings in pilot study, in this main study the results of the initial pre-test showed no statistically significant difference between the experimental and control group of subjects regarding their previous knowledge of computer programming.

Instruments
For the purpose of our pilot study and main study a gamified and non-gamified version of two e-learning modules were developed with the use of the Moodle Learning Management System (LMS).
In the pilot study, two versions of online learning materials on the topic "Lighting and Rendering" were created in Moodle LMS for the university course "3D Modeling." The gamified version of this e-module used a number of special plugins for Moodle which enabled the use of various elements of computer games that were not available in the initial Moodle version at the time of the pilot study. The gamified version of the Moodle LMS was tested before placing the newly created educational material in its virtual learning space. This gamified version was used for the experimental group of subjects. A standard version of the Moodle LMS run on a different server was used for placing equal educational material (but without gamification elements) for the control group of subjects. Both versions of Moodle enabled tracking of students' activities in the system based on Moodle logs. One of the goals of the pilot study was to test the gamification-related technological solutions and resolve technical problems before engaging more research subjects in the main study. The newly created educational content was placed in the Moodle LMS after both systems had been fully tested and became operational. The basic educational content was in the form of HTML text with illustrations and video material. The topics of this educational material were not a part of in-class face-to-face lectures or any exercise that took place in the computer laboratory.
In the main study, two versions of online learning materials on the topic "Heap and Stack" were created in the Moodle LMS for the university course "Programming II." As in the pilot study, the gamified version of this e-module used numerous plugins which enabled the application of various elements of computer games that were not initially available for Moodle 2.7 (see Table 1). The gamified version of the Moodle LMS was used for the experimental group of subjects. The standard version of the Moodle LMS was used for placing equal educational material (without gamification elements) for the control group of subjects. Again, both versions of the Moodle LMS enabled the tracking of students' activity. The basic educational content was in the form of HTML text with illustrations and video material. As in the pilot study, the topics of this educational material were not a part of in-class face-toface lectures or any exercise that took place in the computer laboratory.
It must be noted that the order of sub-topics and the textual, visual and video components of respective educational materials in the gamified and non-gamified versions of the Moodle LMS in both the pilot study and the main study were equal. A pre-test (with 32 items) and post-test (with 32 items) were designed for the educational topic "Lighting and Rendering" in the pilot study. Also, a pre-test (with 30 items) and post-test (with 31 items) were designed for the educational topic "Heap and Stack" in the main study. In both studies the pre-test was administered before the students were given access to the gamified and non-gamified version of the e-learning module "Lighting and Rendering" in the pilot study and "Heap and Stack" in the main study. In both studies it was found that, according to the pre-test results, there was no statistically significant difference between the experimental and control group of subjects regarding the average scores in the pre-test.

Research Procedure
The first phase of our study started with an extensive literature analysis on the topic of gamification, with a special emphasis on investigating possible computergame-related elements that could be implemented in an online course. A conceptual model of gamification elements that are suitable for e-learning courses was developed according to the work of authors like (Schonfeld, 2010), (Deterding et al., 2011;2012), (Werbach and Hunter, 2012) and (Chou, 2015).
In the second phase of our study the Moodle LMS was adapted according to the possible gamification elements that were (a) already present in Moodle 2.7, or (b) available as plugins for Moodle. In the pilot study, a gamified and a non-gamified version of the educational material were developed for the e-learning module "Lighting and Rendering." The gamified Moodle LMS had been tested by the researchers before use by the experimental group of subjects.
During the third phase of our study a pre-test was applied to the experimental (N = 28) and control group (N = 27) of subjects in the empirical pilot study. The subjects were students who attended the university course "3D Modeling". Both groups of subjects were given access to respective variants of the e-learning module on the topic "Lighting and Rendering" for approximately 2 weeks. Finally, a post-test was applied in both groups of subjects. Data analysis was performed regarding the results of the experimental and control group in the post-test as well as in relation to their activity in the Moodle LMS during the time they had access to the online educational material.
The fourth phase of our study was the empirical main study. A pre-test was initially applied to the experimental (N = 99) and control group (N = 102) of subjects. The subjects were students who attended the university course "Programming II". After the pre-test, both groups were given access to a respective (i.e. gamified or non-gamified) version of the e-learning module on the topic "Heap and Stack" for approximately 2 weeks. After that, a post-test was applied to both groups of subjects. As in the pilot study, data analysis was performed regarding the results of the experimental and control group in the post-test as well as in relation to their activity in the Moodle LMS.
The results of data analysis for the pilot study and main study as well as the discussion of results are included in the continuation of this paper.

Results
To ensure that the experimental and control group of subjects in the pilot study are equivalent regarding their prior knowledge, a pre-test was applied with items related to the subjects' overall knowledge of 3D computer graphics. The results that are presented in Table 3 indicate that there was no statistically significant difference (t = 0.48, p>0.05) in the average prior knowledge between the experimental (GE) and Control Group (GC) of subjects.
Likewise, the data presented in Table 3 confirm that in the main study there was no statistically significant difference (t = 0.57, p>0.05) between the experimental (GE) and control group (GC) regarding their prior knowledge of programming. These students had previously completed the introductory course "Programming I", so the pre-test included questions regarding variables, logical operators and C++ commands.
The students in the "3D Modeling" university course in the pilot study used the online content of the gamified and non-gamified Moodle e-module "Lighting and Rendering" for two weeks. A post-test was then applied to test the first hypothesis (H1). The data presented in Table 4 indicate that there was a statistically significant difference (t = 3.99, p<0.01) between the average posttest score of the experimental (GE) and control (GC) group of subjects (p<0.01).    In the main study, the students in "Programming II" university course used the online content of the gamified and non-gamified Moodle e-module "Heap and Stack" for two weeks. A post-test was then applied to measure their knowledge of that topic and thus test the H1 hypothesis. According to the data presented in Table 5, there was a statistically significant difference (t = 3.08, p<0.01) between the average post-test score of the experimental (GE) and control (GC) group of subjects in the main study.
Based on the data presented in Table 4 and Table 5, it can be concluded that the first hypothesis H1 was confirmed both in the pilot study and in the main study. In other words, our study confirms that an elearning course that is designed with the use of elements of computer games will result in greater student achievement regarding test results in comparison to the use of a traditional non-gamified e-learning course. The results of our study in relation to the second hypothesis (H2) are presented in the continuation of this paper.

Analysis of log data on students' online activity
The Moodle LMS enables the tracking of students' online activity regarding their access to online course content. These logs can be used to test the second hypothesis (H2) that the use of an e-learning course with elements of computer games will have a positive effect on the frequency of use of teaching material in this course.
In the pilot study, the "Lighting and Rendering" elearning module in the Moodle LMS consisted of 12 components of online learning content and bonus teaching materials that were identical in the gamified version of this e-module for the experimental group of subjects and in the non-gamified version of the e-module for the control group of subjects. Also, in the main study, the "Heap and Stack" e-learning module in the Moodle LMS comprised of a total of 10 components of online learning content and bonus teaching materials in both versions of the e-learning module. In both studies the educational material was available to students for at least 14 days.
The results of log data analysis from the Moodle LMS for the experimental and control group of students in the pilot study are presented in Fig. 3. The part of the graph in Fig. 3 that is related to the frequency of access to the 12 components of the learning content of the "Lighting and Rendering" e-module by the students in the experimental group indicates a high average rate of access (between 3 and 20 times per student). However, the frequency of access to equal learning content of this e-module by the students in the control group was much lower and indicates that in most cases the students did not access these materials at all. This could be explained by their lack of experience in the use of e-learning module, insufficient interest, acquisition of the learning material (PDF documents) from other students (off-line), or by their reliance on previous knowledge. Furthermore, this non-gamified version of the e-module lacked enhanced visual appearance and included only a limited number of elements associated with computer games like avatars, forums and non-linear access to materials (see Table 1). It can be concluded that the data presented in Fig. 3 indicate a much greater engagement (regarding access to learning material) of students who were assigned to the experimental group and used the gamified version of the e-module.
The analysis of log data from the Moodle LMS was also used in the main study. These logs are presented in Fig. 4. Similar to the results of the pilot study, the part of the graph in Fig. 4 that is related to the frequency of access to the 10 components of the learning content of the e-module "Heap and Stack" by the students of the experimental group indicates a much higher average rate of access (between 2 and 4 times per student). On the other hand, the frequency of access to equal learning content of this e-module by the students in the control group was comparable to the number of students in this group in at least 50% of the learning material. Again, the explanation of the lower rate of access to learning material in the control group cannot be restricted to the fact that their version of the e-module was not gamified, but should also include factors such as a potential lack of interest, prior knowledge of the learning topics, or exchange of the learning material (PDF documents) between students (without the need to access the material from within the e-module). It must be noted that the nongamified version of the "Heap and Stack" e-module also included a much smaller number of elements associated with computer games like avatars, forums and non-linear access to materials (see Table 1). As in the pilot study, the data presented in Fig. 3 indicate a much greater frequency of access to learning material by the students who were assigned to the experimental group and used the gamified version of the e-module. However, it must be emphasized that the students of the "Programming II" university course in the main study had previously used the Moodle LMS for this university course as well as for numerous other university courses at their college.
In our main study the average difference in the level of activity was almost 5 times higher in favor of the experimental group of respondents. The data collected from the pilot study and main study that are presented in Fig. 3 and 4 support the conclusion that the second hypothesis H2 was also confirmed. Therefore, our study also confirms that the use of an e-learning course with elements of computer games has a positive effect on the frequency of use of teaching material in such a course in comparison to the use of an equivalent e-learning course with basically the same teaching material but without the use of computer games elements.

Conclusion
In our research we performed a pilot study and a main study on the effects of gamification of e-learning courses on the achievement and motivation of students of programming. To test the related hypotheses, gamified and non-gamified versions of two e-learning modules (on the topics "3D Modeling" and "Heap and Stack") were developed. The gamified versions included numerous elements of computer games that were implemented in the Moodle LMS, the initial version of which was extended with plugins. Both studies used an experimental and control group of respondents and confirmed positive effects of gamification on student achievement.
In the pilot study, the average number of points measured by the post-test for the experimental group was 36.5% greater in comparison to the control group of subjects (see Table 4). Also, in the main study, the experimental group exceeded the control group by 18.0% regarding the average post-test score (see Table  5). Both differences were statistically significant at the p<0.01 level and led to the acceptance of the first hypothesis (H1) that an e-learning course that is designed with the use of elements of computer games will result in greater student achievement regarding test results in comparison to the use of a traditional nongamified e-learning course.
In a study by Fotaris et al. (2016) the experimental group of students in a computer programming course manifested a greater number of downloads of reference material in comparison to the control group. In our study we obtained a similar result in both the pilot study and main study. In the pilot study, 12 teaching and bonus materials were provided in the gamified and nongamified version of a two-week e-module "Lighting and Rendering", while in the main study 10 teaching and bonus materials were provided to students on the topic "Heap and Stack." As can be observed in Fig. 3 and 4, the experimental group of students accessed the learning materials much more frequently, which can be interpreted as an indirect manifestation of greater motivation for learning.
Having in mind our investment in creating a gamified learning environment for our pilot and main study with numerous game-based elements/activities, it should be noted that other authors have reached a supporting conclusion in that respect regarding the learning of C programming language (Ibáñez et al., 2014): "In view of the encouraging results in relation to students' engagement toward the academic activity presented, we conclude it is worth designing and evaluating more gamified experiences." Does such an investment of instructors' time and effort in gamifying an online course pay off in greater learning achievement? Again, we would like to cite a conclusion by other authors in relation to learning parallel programing, which points in a similar direction as that of the authors of this paper (Fresno et al, 2017): "The application of gamification mechanisms implies a higher effort done by the course instructors, compared to classical master lessons. However, in light of the results of the studies we have presented, we believe that it is worthwhile to pay the cost of applying these techniques." Finally, one of the intentions of the authors of this paper was to develop mechanisms for providing feedback to students that would facilitate their extrinsic motivation for learning. As a matter of fact, with regards to implementing gamification in another software development course, the authors of a corresponding study have reached the following conclusion (Fotaris et al., 2016): "Coupled with effective pedagogy, games can offer a more effective and less intrusive measurement of learning than traditional assessments".

Limitations
Research activities for each course in our pilot study and main study were specifically planned with respect to the course curriculum and the students' workload related to other parallel courses. It was decided that the knowledge measurement should be carried out within approximately 2 weeks from the moment when the students were able to access the learning materials in the Moodle LMS. Therefore, there is a limitation regarding possible generalizations of the results of our study to online courses which last a full semester or even longer.

Author's Contributions
Andrija Bernik: The main responsible author for research design, design of gamified e-courses and implementation of pre-research. Contributed in planning, design, writing, data collection and analysis of the results.
Danijel Radošević: The main responsible author for implementation of the main research. Contributed in design, planning, data collection and analysis of the results.
Goran Bubaš: The main responsible author for literature review and writing. Contributed in literature review, writing, translation and analysis of the results.

Ethics
This article is original and contains unpublished material. The corresponding author confirms that all of the other authors have read and approved the manuscript and no ethical issues involved.