Trade-Off of Frame-Rate and Resolution in Online Game Streaming

: Only limited research in the literature focused on assessment of different spatial and temporal resolutions. In addition, the field of synthetic video compression, such as video compression in game streaming, was the focus of only a few research papers. This paper will consider a wide range of synthetic video textures and motion speeds. The visual quality of the compressed video sequences is perceptually evaluated using the Double Stimulus Impairment Scale (DSIS) assessment metric. Extensive subjective viewing tests to evaluate user satisfaction are performed. A methodical analysis to study the influence of different dimensions on Mean Opinion Score (MOS) is presented. The study considers a wider scope of dimensions that affect the perceptual visual quality. MOS results are shown in details for different bit rates, frame rates and frame resolutions. This study contributes novel ideas that should provide common rules for online game streaming over transmissions with relatively low bit rates.


Introduction
Online gaming is a common type of entertainment nowadays. Computer online gaming has grown to become one of the largest commercial investments as it brings in substantial revenues . The fast growth of communication and computer networking has given more flexibility to application providers and enabled them to increase the service quality of their products (Claypool and Claypool, 2007;Claypool, 2009). Online game creators constitute one sector that developed its applications to exploit this advancement. As a result, streaming videos of newer online games consume more bandwidth and require higher bit-rates. Therefore, video up-to-date compression techniques are required to reduce the bitrate into appropriate levels without compromising the quality (Claypool, 2009).
The main idea of video compression in online game streaming is to maximize the amount of data transmitted over the network within the bandwidth limitations. Thus, the main benefit of a video codec is the ability to reduce video size while conserving quality and fidelity. Video codecs vary in compression capability. However, there are many parameters that affect the process of video compression; mainly the frame rate, bit rate and frame resolution (Claypool et al., 2006;Abdalla et al., 2014).
An online game player needs time to interact with other players by viewing their screens. This is achieved by adding extra screens for each member, which makes small and medium resolution frames more desirable. Several available online platforms, such as (http://www.xfire.com; http://www.twitch.tv), allow users to stream the videos of their game layout.
Video compression plays a very important role in game streaming over the Internet, where bandwidth limitations are significant factors to be taken into consideration when such video game streaming platforms are designed. Consequently, there is a need to focus on video game streaming compression.
Online game providers try to offer the best user entertainment during game-playing. They focus on user needs and try to provide high quality service. However, bandwidth increases are limited and not always obtainable (Claypool and Claypool, 2007a). In addition, the cost of bandwidth increase is an important factor in commercial applications. The main challenge is to meet customer expectations in spite of the limitations of available resources. Since higher quality implies higher cost, service providers need to identify the minimum levels of quality that provide user satisfaction.
Subjective video quality measurements depend on the perceived quality of video sequences according to the assessment of observers . The double stimulus  impairment scale variant II (DSIS II), which was  suggested by an ITU-R recommendation (ITU-R, 2002), is a subjective assessment metric that will be used in this study. For completeness, different spatial and temporal resolutions will be considered.
For video game streaming, a study of the trade-off between frame rate and resolution is recommended for each bit rate. A good trade-off is expected to provide clear and smooth motion of the game components during a game play and to provide appropriate fidelity for team members within available bandwidth. Increasing frame rate and frame resolution will increase the overall video quality; however, this will be limited where bandwidth limitations restrict game broadcasters to stay within certain parameters ).
Consequently, this paper will present recommendations for the trade-off between frame rate and resolution to provide higher user satisfaction within typical uplink bandwidths. In addition, this paper will show that video smoothness, indicated by frame rate, could be more significant and effective than frame resolution. Recommendations for frame rate and resolution at each bit rate will be presented.
The rest of this paper is organized as follows: Section 2 summarizes the related works; Section 3 describes the methodology of implementing the subjective test, testing environment and results; Section 4 provides statistical analysis of obtained results; and Section 5 summarizes the outcome of the paper and suggests future work.

Related Works
Although many studies have been done on video codec comparison, previous research did not focus on video codec capability in synthetic video compression and streaming. Abdalla et al. (2014), we studied the compression performance of three of the most popular encoding techniques. The study focused on the codecs capability for synthetic videos as video gaming became more attractive for users during last few years. As an extension of the previous work, this work presents a wider study considering the latest video coding techniques such as VP9 and HEVC. In addition, the limitations of network resources are considered and trade-off between frame rates and resolutions has been taken into consideration. Our previous work did not discuss that where best trade-off values are suggested in this work to provide maximum user satisfaction with exploiting limited available resources.
The study of computer graphic sequences and the search for the best trade-off between frame rate and frame resolution within typical uplink parameters lacks sufficient research (Wang et al., 2013). However, some recent research proposed new techniques to optimize spatial and temporal resolution based on subjective evaluation (Takagi et al., 2014).
Cloud gaming has recently become reality as many game developers focused on cloud computing to improve online gaming. Cloud gaming is based on rendering the game remotely on the cloud and streaming scenes back to the users as video sequences (Shea et al., 2013).
A study of the effects of frame rate and resolution on user performance in computer games was presented in (Claypool et al., 2006). The performed study focused on the First Person Shooter game and concluded that frame rate has a significant impact on player performance and enjoyment. A study of the effects of resolution on users playing First Person Shooter was presented in (Claypool and Claypool, 2007). The effects of resolution in high and low contrast, in addition to full screen and windowed display, were considered in the study.
The study of algorithms for rate control is an active research topic. Studies usually focus on a specific codec and try to improve it, or they propose techniques for exploiting network bandwidth (Smets and Overbeeke, 1995). Smets and Overbeeke (1995), a study of frame rate and resolution effects on the most common game actions, such as shooting and navigation, was presented. The study concluded that frame rate has a much greater impact on game actions than frame resolution. The relative importance of spatial resolution, temporal resolution and intensity resolution factors was also presented in (Smets and Overbeeke, 1995). The study concluded that the interactive task is highly affected by the temporal resolution, where spatial resolution is much less important.
Several studies of video codecs for real videos were conducted. However, these studies did not focus on codec efficiency with synthetic videos. Deng et al. (2010), a comparison study between H.264 and Motion JPEG2000 for high definition video coding was conducted. Wiegand et al. (2003), the performance of H.264, MPEG-4, H.263 and MPEG-2 was studied. These codecs were compared using PSNR and subjective testing schemes. An evaluation of perceptual visual quality under various settings and requirements was conducted in (Zhai et al., 2008). The subjective assessment tests were analyzed to study the influence of the different dimensions on the subjective evaluation. The dimensions used were encoder type, video content, bit rate, frame resolution and frame rate; where H.264 and H.263 were the codecs used in the study. Ya-Fan et al. (2009), an adaptive media playout scheme was proposed. The proposed scheme tries to keep video playout smoothness as high as possible while adapting to channel conditions. McCarthy et al. (2004), a methodology was presented to evaluate perceived video quality when watching high motion videos. The video sequences were chosen from some football matches with different physical quality metrics. 720p and 480p frame resolutions were used to study the relationship between frame rate and quantization and their impact on perceived quality. Tan and Chou (2012), a framework is proposed to adjust encoder frame generation and decoder playout frame rate. The main idea focused on reducing prebuffering requirements while maintaining video streaming continuity. Claypool and Claypool (2007b), a classification of actions in First Person Shooter games was provided. The impact of frame rates on player performance was evaluated by qualitative assessment supported by quantitative analysis of user studies. Chuah et al. (2014), a novel layered coding scheme was presented. It controls the increase of graphical processing capabilities to reduce the bit rate of game streaming of a mobile client.
Many techniques have been proposed to improve and develop video games. Algfoor et al. (2015), popular path finding techniques and algorithms on video games have been summarized, where the effect of such techniques and algorithms on video games are correspondingly discussed and explained. However, it did not focus on the effect of video encoding on game-player experience. On the other hand, this proposed paper does not deal with the internal algorithms of a video game and it studies the game as a video stream where it searches for the encoding mechanism that provides the best gameplayer experience. The game is considered as a video sequence that has been rendered and made ready for streaming. The goal of this proposed study is to find the minimum encoding cost of a video game without compromising game player satisfaction.
One major innovation in the field of video compression is the H.264/MPEG-4 Advanced Video Coding (AVC) standard (Wiegand et al., 2003b). Although the H.264/MPEG-AVC codec achieved about 50% increase in coding efficiency compared to earlier codecs, many researches have been proposed to improve its encoding time and efficiency (Mazhar and Abdalla, 2016).
More recently, the H.265 standard, also known as High Efficiency Video Coding (HEVC), has been introduced. A study presented in (Seeling and Reisslein, 2014) found that H.265 has superior performance compared to H.264 in terms of rate distortion and network-link bit rate. However, H.265 is more computationally expensive to encode at the same speed as H.264 due to its larger prediction units and expensive motion estimation. Experimentally, H.265 required higher encoding time than H.264 (Mazhar, 2016). Furthermore, it was found that H.265 encoding time was higher than VP9 encoding by a factor of 7.35 on average, where VP9 encoding time was more than 100 times higher than that of the x264 version of H.264 (Grois et al., 2013).
Consequently, HEVC generally requires a built-in hardware decoder to substitute hardware cost for software cost. Even though this extra cost is reasonable for ultra-HD video applications, it may not be practical for applications of lower resolutions, such as those of popular online games.
Overall, H.264 has shown reliable, fast and efficient compression capabilities and it has been adopted into many applications. In addition, it remains less expensive than more recent codecs while providing sufficient performance for online gaming and for a variety of other applications. Therefore, it was chosen to be used in the implementation of this study.

Subjective Viewing Test
To perform a subjective viewing test, three testing aspects are considered; namely the materials, method and environment. Then, test results will be represented with the Mean Opinion Score (MOS).

Testing Materials
Four ten-second test video sequences are viewed by thirty viewers. The compression experiments are applied over diverse types of captured video sequences in order to provide different scene contents. The scenes include sequences with a range of High-Texture High-Motion (HTHM) containing very fast action movements of avatars and a lot of communicated written messages between players, High-Texture Low-Motion (HTLM) containing many written messages on the screen but with slow action movement of avatars, Low-Texture High-Motion (LTHM) containing no written messages but with high object movement and need for quick player response and Low-Texture Low-Motion (LTLM) that contain mostly one object with slow motion, such as a monster only moving its head with a mono-color background.
The sequences used to perform this comparison study were captured from the game "League of Legends" (http://eune.leagueoflegends.com). This game is considered to be one of the most popular online games. It contains a very wide range of features and visual specifications. The sequences were captured using an expert capturing software called FRAPS (http://www.fraps.com). This free professional software was used to record videos while the game was being played.
Considering the above four different video sequence types with different combinations of bit rates, frame rates and frame resolutions, a total of 144 different scenarios were examined. To avoid user exhaustion because of the long watching time, viewers were asked to do their evaluation in four sessions on four different days. All video sequences were compressed using an H.264 video encoder with bit rates 512, 1024 and 2048 kilobits per second (kbps) and frame resolutions 480, 720 and 1080p, while frame rates ranged from 15 to 30 frames per second (fps). These bit rates and frame resolutions were chosen because they are typically used in online game streaming.
To conduct a meaningful subjective evaluation, all video sequences were displayed at the same resolution and frame rate; which was frame resolution 1080p and frame rate 30 fps. Lower resolutions were up sampled to 1080p and frames for lower frame rates were repeated to raise the rates to 30 fps. The process of up sampling and frame repetition was conducted using the H.264/AVC 6-tap half-sample interpolation filter and frame repeat (ITU-R, 2005).

Testing Method
All video sequences were captured with a length of ten seconds. A variety of motion speeds and textural contents was considered in the capturing process. Five levels of quality scales, using subjective DSIS II, were conducted in this evaluation study. The method displays the original non-compressed sequence followed by the compressed one. This process is repeated once before the viewers can score the perceptual quality. The viewer perceptual quality is scaled in five levels describing the video quality score. The scores are 1, 2, 3, 4 and 5, which represent 'bad', 'poor', 'fair', 'good' and 'excellent,' respectively.

Testing Environment
The laboratory specifications were prepared according to the ITU-R recommendation (ITU-R, 2002). The experiments were conducted in a computer lab using19-inch Dell LCD monitors with videos shown in full-screen mode. The distance from viewers to the monitors was set to three times the display-screen height. All participants performed this evaluation at the same time, but separately without seeing each other's screens. The lab windows were covered with gray curtains and white lights were used for lighting. All viewers used the same lab and monitor specifications. All 30 viewers were B.Sc. students with suitable communication skills and good computer gaming experience. Additionally, the scaling and scoring mechanism was explained to the viewers to ensure vote reliability.

Test Results
MOS results are shown in Fig. 1, 2 and 3, where the following observations can be obtained. At the 1080p frame resolution, illustrated in Fig. 1, the average MOS increased when bit rate increased. Thus, for a fixed bit rate, MOS mostly decreases when frame rate increases. At higher bit rates, the MOS decrement was more than the decrement at lower bit rates. For the 720p resolution, illustrated in Fig. 2, results showed that MOS increased when bit rate increased. The MOS increment increased when bit-rate was raised from 512 to 1024 kbps, which was less than the increment when bit rate was raised to 2048 kbps. Hence, MOS improvement due to bit rate increase at higher bit rates was less than that at lower bit rates.
The results of the 480p frame resolution, illustrated in Fig. 3, showed that MOS values at all bit rates were nearly similar.

Analysis of Results
In order to methodically study the effect of frame resolution and frame rate on MOS, Analysis of Variance (ANOVA) is used to conduct this evaluation. A threeway ANOVA was adopted to test the effect of Video Type (VT), Bit Rate (BR) and Frame per Second Rate (FpS) for different frame resolutions on MOS.
ANOVA is a statistical technical method that can be used to compare the means of two or more groups under certain assumptions. (The assumptions will be discussed later; when the MOS model is presented). Several types of ANOVA are available in the literature and can be used in comparisons, such as one-way ANOVA, twoway ANOVA, factorial analysis and other types. To test the effect of VT, FpS and BR on MOS, three-way ANOVA is adopted in this analysis.
The ANOVA method was performed on MOS data obtained from viewers. This study considers a wide scope of dimensions and frame rates that affect the perceptual visual quality. In preceding sections, the results of subjective view tests for assessing the perceptual quality were presented. MOS results were shown in details for different bit rates, frame rates and frame resolutions. The analysis in this section will focus on capturing any difference in MOS mean under the above potential factors. The MOS model that will be used in the analysis is described in Equation 1: The dependent variable in this model, MOS, is assumed to be influenced by at least one of the independent variables with some random error, ε. In other words, the variation in MOS values is attributed to the use of different variations of the independent variables. It should be noted that the error in this model, ε, is assumed to be normally and identically distributed with mean 0 and variance σ2. The results of ANOVA analysis with 480p are shown in Table 1. Each column shows a specific output and can be explained as follows. The second column is the sum of squares between treatments of each dimension. The third column is the degrees of freedom related to each dimension model, which can be calculated as (number of treatments -1). The fourth column shows the ratio of sum of squares to degrees of freedom. The fifth column is F statistic and the last one is the p-value. The p-value indicates the significance of the dimension and can be derived from the cumulative distribution function of F.
As shown in Table 1, the p-value was less than 0.05 for both dimensions, FpS and VT and it was greater for BR. This indicates that MOS values were affected by FpS and VT significantly. However, BR was considered not to affect MOS since the corresponding p-value was insignificant (p≥0.05).
Since the p-value for each of FpS and VT was (p≤ 0.05), this is an indication that these two dimensions substantially affected MOS. However, the small pvalue of VT, which was almost zero, implies that MOS was significantly affected by this dimension, where MOS was affected slightly less by FpS than VT. The pairwise comparison study of frame rate at the 480p resolution is illustrated in Table 5. As shown in the second column of the table, the mean difference magnitude indicates that FpS value (I) affected MOS better than FpS value (J) when it was positive and worse when negative. The results show that increasing frame rate from 15 fps to higher rates increased the MOS, but it is clear from the p-value that increasing frame rate was most significant at 20 fps. Increasing it more than 20 fps was less significant and this means increasing frame rate to more than 20 fps will worsen the MOS in addition to causing an increase in bandwidth consumption.
Therefore, at such low spatial resolutions, it is not always true that increasing the frame rate leads to an increase in the perceptual quality. This claim is clearly shown by comparing the results obtained with 15 fps to those with other frame rates. The mean difference values showed that 20 fps at the 480p resolution gave better MOS than higher frame rates.
In the case of comparing results at 30 fps to those with other frame rates, it can be observed that increasing frame rate from 15 to 30 fps was significant (p≤0.05) and highly affected MOS, where it was insignificant at 20 and 25fps. The full pairwise comparisons are shown in Table 5. Results of the 720p resolution are shown in Table 2. The results show that the p-value was borderline significant for VT, where it was insignificant for BR and FpS. This implies that MOS at the 720p resolution was slightly affected by changing the video type, but the effect of changing bit rate and frame numbers on MOS was negligible.
The p-value magnitudes were 0.397 and 0.885 for BR and FpS, respectively. Here, the first value was closer to the significance threshold (0.05) than the second one. This indicates that BR could affect MOS more than FpS if they were considered.
The pairwise comparison for the 720p resolution is shown in Table 6. Results show that increasing frame rate from 15 to 20, 25 and 30 fps was insignificant (pvalue ≥ 0.05). In addition, the results show that increasing frame rate from 15 to 20 or 25 fps worsened the perceptual quality. By comparing the results obtained with 30 fps to the results with other frame rates, it can be seen that increasing the frame rate to 30 fps enhanced the perceptual quality. However, this enhancement is considered insignificant. Table 3 shows the results obtained with the 1080p resolution. It can be concluded from the p-values that all dimensions were insignificant. The results indicate that VT, BR and FpS affected MOS insignificantly. Nevertheless, according to the magnitude of p-values, it can be seen that BR relatively affected MOS result the most, where FpS affected it relatively less and the least relative effect was of VT.
It can be concluded from Table 3 that the 1080p video sequences were less affected by the diversification of video types where it had the least effect on such resolutions. However, the effect increased at the 720p and 480p resolutions. On the other hand, the BR effect increased when the frame size was increased. It showed a higher effect at 720p where the highest effect was at 1080p. Table 7 shows the pairwise comparison of frame rates at the 1080p resolution. It can be seen that the increase of frame rate increased the perceptual quality. However, the increase is considered insignificant since (p≥0.05). The results showed that any increase of frame rate would have a positive effect on MOS. Mean difference results showed that higher frame rates provided higher MOS. The detailed comparison values are provided in the table.
As it can be observed from the above result analysis, changing frame rate or frame resolution affects MOS in different ways. In some cases, changing frame rate slightly affects MOS considerably and vice versa. On the other hand, changing frame resolution has substantial effect on MOS in some cases and a lower effect in other cases.      The extensive experiments conducted in this research indicated that a clear criterion should be used to find the frame rate and frame resolution most suitable for use with a given bit rate while preserving gameplayer satisfaction. A trade-off between frame rate and frame resolution is the chosen criterion for providing the best satisfaction within an available transmission bit rate. The above result analysis has studied the effects of different elements on MOS results at different resolutions. This leads to the final important contribution of this paper, which is presenting the recommended trade-offs between frame rate and frame resolution. The recommendations for each bit rate with different video types can be construed as in Table 4.

Conclusion
In this study, the effects of frame rate and resolution on perceptual quality were studied. The study was conducted using extensive subjective viewing tests using DSIS II. The frame rates and resolutions were tested for low bit rates used in uplink streaming typical at the present time. The results and their statistical analysis have led to new interesting perceptions on online video-game streaming. First, the 480p video sequences were highly affected by video type and frame rate, where they were insignificantly affected by bit rate. The video type affected sequences with 720p and 1080p resolutions in descending order. Second, perceptual quality was affected significantly by frame rate at the 480p resolution. However, it was insignificantly affected at higher resolutions such as 720p and 1080p. Third, for the lower bit rates tested, the higher frame resolution with lower frame rates was more acceptable by observers especially for high texture sequences.
Subsequently, analysis of these results led to frame rate and resolution recommendations for each bit rate. At a bit rate of 512 kbps, a low frame rate such as 15 fps is recommended with a resolution of 720p for low texture and 1080p for high texture sequences. At higher bit rates, such as 1024 and 2048 kbps, the 1080p resolution is acceptable. Nonetheless, a frame rate of 15 bps is only acceptable for lowmotion sequences at these bit rates, where a frame rate of at least 20 fps is needed for high-motion sequences. Naturally, a reasonably higher frame rate can be used when the bit rate is increased.
Overall, the results of our research should provide common rules for online game streaming over transmissions with relatively low bit rates. In addition, applying this study to game and video streaming in small mobile devices; i.e., smart phones and pads, should be attempted and studied. Such devices generally have smaller screens and lower capabilities than personal computers and they require lower screen resolutions.
Further future work may study online streaming of animated movies by partitioning a synthetic video sequence into segments according to texture and motion where some segments are down-sampled and have their frame rates reduced according to the recommendations of this paper. Then, the effect of this down-sampling and frame rate reduction on perceptual visual quality and continuity of the whole sequence should be evaluated.

Author's Contributions
Ahmad Abdel Jabbar Ahmad Mazhar: Made significant efforts on understanding the problem and preparing suggested solutions. He contributed to analyze the chosen solutions, running experiments, gathering data and data analysis and interpretation. He also helped in drafting and reviewing the article and approved on the final version to be submitted for publication.
Ayman Mahmoud Aref Abdalla: Made clear efforts on suggesting solutions for the problem, analyzing the suggested solutions and recommending one to be used. He also ran experiments, analyzed results, interpreted gathered data and helped in drafting and reviewing the article. He also made critical review on approving the final version and added significant content to the article.

Ethics
This article is original and contains unpublished material. The corresponding author confirms that all of the other authors have read and approved the manuscript and no ethical issues involved.