By Dr Yiannis Andreopoulos and Dr Vasileios Giotsas, iSIZE Technologies
As discussed in recent interviews at edie.net and techerati, video is now beginning to cause major spikes in IP network traffic, energy consumption of data centers, and battery usage of end-user devices. With regards to network traffic, well-established mechanisms, like DASH, HLS and CMAF, handle network bandwidth fluctuations and bandwidth limitations by allowing for clients to switch to lower-bitrate and (typically) lower-resolution versions of the video during playback at the expense of quality-of-experience (QoE) degradation.
With respect to battery usage, screen dimming mechanisms that optimize energy consumption and battery life in mobile devices can also cause significant QoE degradation in video playback. It is surprising that there has been little attention paid to the impact of screen dimming on QoE of video streaming, especially considering the volume of video content consumed daily by mobile, tablet, laptop and Smart TV users.
Problem Statement
In all modern operating systems, ambient light sensors and battery monitoring tools constantly inspect and adjust screen brightness in order to ensure it is adjusted to the ambient light conditions, while also reducing energy consumption and prolonging battery life. This is because, as shown by several studies, the bulk of the energy consumption of smart TVs, portable players and mobile phones stems from data transfers, decoding and displaying video.
Alas, as shown by the left part of Figure 1 below, the typical strength of screen dimming occurring during energy-saving modes does not allow for reasonable quality-of-experience during video playback. The situation is so bad that many users of video applications disable adaptive screen dimming features in order to be able to watch content without severe QoE degradation, albeit at the expense of battery life of their device.
Figure 1. Left: Example of video playback under screen dimming for energy saving. Right: Video playback under the same screen dimming conditions when using the iSIZE deep video precoding optimized for energy-saving mode adaptation.
What one would like is for the video to auto-adjust to the screen dimming conditions, i.e., when detecting a client is switching to energy-saving/dimmed-display settings, reverse-amplify content brightness in a bitrate and scene-adaptive manner in order to alleviate the visual effect of screen dimming. An example of this is shown on the right part of Figure 1 above, where the same content is enhanced at the sender side to compensate for the visual effect of screen dimming at the client, thereby showing a dramatic visual difference to the content of left side. Beyond improving QoE, given that screen dimming masks certain details of the content, it may also be possible to save bitrate during streaming, which is currently spent unnecessarily when streaming content to devices set to low display brightness levels.
Solution: Screen-Dimming Aware Video Precoding
Given this situation, we investigate the impact of our intelligent video preprocessing technology in optimizing for video playback on battery-power laptop and mobile devices that support energy-saving modes.
Our design is as follows:
- We customize our bitrate-saving video precoding modes from bitsave.tech platform by training them to pseudo-reverse screen dimming features found on typical energy-saving modes of devices; this typically entails training our models assuming there is certain brightness, contrast and gamma adjustment at the client display, and learning to correct for that while at the same time retaining bitrate as low as possible and enhancing visual quality.
- After pretraining, we are able to deploy our precoding modes both for bitrate saving, as well as for adaptation to screen dimming conditions. An example of the different modes is given in Figure 2, where we show that it is possible to correct for screen dimming conditions and improve the displayed video quality while offering 30% saving in bitrate in comparison to the original bitstream.
Figure 2. Illustration of the available modes during iSIZE screen-dimming aware precoding (menu of options at the bottom right of the figure): ‘OFF’ is original video; ‘OFF – 23% saving’ is iSIZE precoding from bitsave.tech saving 23% bitrate vs. the ‘OFF’ bitstream but without screen-dimming awareness; ‘ON’ is iSIZE precoding done only for screen dimming; ‘ON – 30% saving’ is iSIZE precoding saving 30% bitrate vs. the ‘OFF’ bitstream and optimizing for client screen dimming mode.
Testing: Deployment in Android and Windows Devices and the Google Chrome Browser
In order to test the impact of our solution in practice, we have created two streams. First, you can examine the visual impact of our approach in a left-right comparison (akin to that of Figure 1) by playing the video below or click here to view in a separate window.
If you are using a battery-powered device running either Windows or the Android OS and playing the video via the Chrome browser, you can try plugging and unplugging your device from its power supply to see the impact in quality versus the dimming of your device’s screen that will be enabled when it goes on battery power. Secondly, if you would like to see the full video and select modes for the entire video playback akin to those of Figure 2, then play the video below or click here to view in a separate window.
As previously, if viewing this video with the Chrome browser, you can also try plugging and unplugging your device from power and observe the change in video playback. Manual selection of modes is also available if you click the cogwheel menu, shown at the bottom right of Figure 2 and manually adjust your screen brightness when switching to the energy-saving mode.
Beyond the visual QoE offered by this adaptation, we measured the battery discharge in two typical consumer devices: a Dell XPS 15 7590 laptop with a 7285 mAh battery, running MS Windows 10, and an Amazon Fire HD 8 tablet with a 3210 mAh battery, running the Fire OS 7 (Based On Android 9.0). We streamed the test videos with the default HTML 5 player of the Google Chrome browser version 83. To log the battery level we used the Javascript BatteryManager API and we listened for levelchange events to capture changes in the battery charging level. To ensure that other applications or background did not run at the same time so that we only measure the battery consumption of video streaming, we stopped every running application and disabled every non-OS background process and service.
In each device, we measured battery discharge levels for two versions of a 2-hour long video, the baseline (original) version and the enhanced (transcoded) version. Both versions were encoded at 540p, 50fps with H.264/AVC (x264 slower preset, VBV encoding) and two different bitrates: The baseline version corresponds to 3500 kbps, while the BitSave screen-dimming enhanced version is at 2450 kbps, i.e., offers 30% saving vs. the baseline. We streamed the baseline version at 100% screen brightness, and the dimming-enhanced version at two different levels of brightness, 50% and 25%. After each playback we emptied the browser cache to ensure that the video will be streamed from the remote server at the next playback and not from the cache. The plots in Figure 3 and Figure 4 show our measurement results for the two different devices and screen dimming settings.
Figure 3: Battery discharge of Fire HD 8. “Enhanced” corresponds to the BitSave screen-dimming enhanced version that is offering 30% bitrate saving vs. the Baseline. The Enhanced-25% reduces the battery discharge rate by 40% vs. the Baseline.
Figure 4: Battery discharge of Dell XPS 15. “Enhanced” corresponds to the BitSave screen-dimming enhanced version that is offering 30% bitrate saving vs. the Baseline. The Enhanced-25% reduces the battery discharge rate by 40% to 50% vs. the Baseline.
In the case of Dell XPS 15 (Figure 4), the impact of streaming the enhanced videos with lower screen brightness is very apparent, with a 40%-50% difference in the battery discharge rate between the baseline and the 25% brightness test. For the Fire HD 8 device (Figure 3), the battery discharge rate difference tends to be somewhat lower due to the smaller screen size, but still remains very pronounced. This battery saving comes while offering the BitSave dimming-enhanced version, which is visually closer to the non-dimmed version of the video encoded without iSize precoding, as you can see from your own inspection of the examples here (for left-right) and here (for the full video).
This first exploration shows that intelligent video precoding can also be used to improve energy efficiency in end-user devices without requiring any changes on the client device. It is very easy to detect the battery level or ambient light sensor level either via a browser plug-in or else via a player, and with this information alone one can provide improved video stream switching at the server side to versions that are optimized for the screen dimming conditions of the client, while offering bitrate saving versus the original video. We look forward to extending these concepts further towards deployment in commercial streaming services.
