Debugging Media in Web Inspector

Media elements, such as <video> and <audio>, can sometimes be tricky to debug due to the sheer flexibility of the available functionality. Content can be loaded in a variety of different ways, such as a direct stream from a single file, byte-range requests from a single file, or multipart streams from different sources. In the past it was difficult to reconcile how content made the journey to being displayed on a page, as well as how it interacted with the other content around it. In macOS Mojave 10.14.4 we’ve added special functionality that makes it easier to understand how media content relates to network and JavaScript activity, specifically how that activity can stall or even prevent playback.

Correlating Media Events and Network Activity

Sometimes media playback may become stalled due to network activity, such as if a byte range request takes too long to load. Previously there was no easy way to identify these issues solely from Web Inspector. Now, however, Web Inspector provides an option called “Group Media Requests” that organizes network activity based on the related top-level media element. This makes it easier to keep track of related resources, such as byte range requests or items that are part of a media playlist.

Since media elements aren’t resources it doesn’t make sense to have the usual “waterfall”. Instead, we’ve created a special timeline that keeps a log of playback related events fired by the media element, such as loadstart or stalled, as well as some basic state about the media element, such as whether it’s paused. A dashed line represents “paused”, a solid line represents “playing”, and no line represents “stopped”. If a <video> element enters fullscreen a grey area is shown indicating the time in which that <video> remained in fullscreen mode. On macOS <video> elements can also be rendered in a special way, called Power Efficient Video Playback, which is indicated by a green area.

Identifying Media Issues Caused by JavaScript

Sometimes, however, issues may arise due to incorrect JavaScript usage or layout issues. For these cases, we’ve also added the same event information to a new timeline in the Timelines tab called Media, as well as in the general timeline overview.

Similar to the Network tab, correlating media events with nearby script activity can be a quick way to diagnose issues in situations where JavaScript causes the wrong activity to occur. In addition to this dedicated Media timeline all media events are also shown in the general timeline overview.

Diagnosing issues with Power Efficient Video Playback

Whenever a <video> element is about to render a frame on macOS, it makes a decision about whether to use the computer’s CPU/GPU or the display itself to composite the frame. In cases where the display can be used for frame compositing, it is often more power efficient. If your website contains full-screen video content, you should strive to follow these guidelines to ensure that your <video> is able to use display compositing as often as possible.

• The <video> must be playing.
• The <video> must be full-screen.
• The <video> must not be occluded in any way (e.g. not showing controls or any content “above” it).

Keep in mind that these are ways in which the web content itself can prevent power efficient video playback. There are other reasons that aren’t controllable from web content as to why a video may not enter a power efficient playback state, such as being connected to an external display or the device itself not supporting it.

WebDriver

Starting in macOS Mojave 10.14.4, WebDriver is able to access the total count of display composited frames which can be used for continuous integration testing.

let video = document.querySelector("video");
let videoPlaybackQuality = video.getVideoPlaybackQuality();
videoPlaybackQuality.displayCompositedVideoFrames

Each time a frame is display composited, that count will increment, so if the count does not increment, the <video> is not using display compositing.

If you develop websites where the main content is a <video>, consider adopting these guidelines to provide a better experience for your users.

Feedback

These enhancements are available starting in macOS Mojave 10.14.4 and iOS 12.2. Is there additional information you’d like us to visualize with respect to media elements? Are there other elements that fit into this “category” for which you’d like to see similar instrumentation? Let us know! Please feel free to send us feedback on Twitter (@dcrousso or @jonathandavis) or by filing a bug.