How to use Media Source Extensions with AirPlay

Media Source Extensions (MSE) is a popular way to provide streaming video on the web. It gives JavaScript control of the way bytes are sent to the browser for playback. At the 2023 Worldwide Developer conference, Apple announced a new Managed Media Source API that improves on MSE with efficient video streaming and longer battery life for iOS and other devices.

However, MMS and MSE, by nature, are not compatible with AirPlay, which requires a unique playback URL. AirPlay allows you to start playback of your favorite videos on your phone, move them to your TV and then switch off that phone. An AirPlay-compatible URL can be of any format such as an mp4, mpeg-ts, or HTTP Live Streaming (HLS).

This post will guide you through providing both sources and, in the process, build out a demo example.

Since MMS/MSE uses binary blobs appended to a SourceBuffer it won’t work with AirPlay. But, if you create an alternative source that can be served as an AirPlay-compatible URL, there is a way to get them to work together.

When it comes to an AirPlay-compatible alternative, HLS is an option that offers a great deal of efficiency for users. There are numerous resources that can guide you in converting your video content to serve it with HLS. Apple offers a toolkit you can use and there are many other options as well.

const airplayURL = './video/demo.m3u8';
const videoURL = './video/demo.mp4';
const mediaType = 'video/mp4; codecs="avc1.640028"';

// Create a video element
const video = document.createElement('video');

// Set video element properties for the demo
video.controls = true;
video.loop = true;
video.muted = true;
video.autoplay = true;

In setting up the MediaSource, it’s easy to use feature detection to use Managed Media Source API to offer power-efficient streaming on browsers that support it and gracefully fallback to MSE where its not available:

// Feature detect MMS and fallback to MSE
const MediaSource = self.ManagedMediaSource || self.MediaSource;
const source = new MediaSource();

We also need a way to offer both our Media Source and AirPlay sources at the same time. The HTML video element’s ability to define multiple sources will do just that. It was originally intended to allow a user-agent to choose the best format of the video to be played and fallback should it not be supported.

<video>
  <source src="format/video.m3u8" type="application/x-mpegURL">
  <source src="format/video.ogg" type="video/ogg">
  <source src="format/video.mp4" type="video/mp4">
  <source src="format/video.webm" type="video/webm">
</video>

The browser will look over the list of available formats from top to bottom. If no matches are found, or if decoding fails along the way, it will select the next element in the list.

We can make use of this behavior, combining the ability for the user-agent to choose the best format and allowing AirPlay to select a playable source. You’ll create a URL from your Media Source and add it to a video element as the first source. This URL is local to the user-agent and can’t be shared, as it has no meaning outside the local context. Then you add the AirPlay-compatible URL as the second source.

// Add MSE/MMS streaming source
const videoSource1 = document.createElement('source');
videoSource1.type = 'video/mp4';
videoSource1.src = URL.createObjectURL(source); // Create URL from MediaSource
video.appendChild(videoSource1);

// Add AirPlay-compatible HLS source
const videoSource2 = document.createElement('source');
videoSource2. type = 'application/x-mpegURL';
videoSource2.src = airplayURL;
video.appendChild(videoSource2);

Now when Safari detects that an alternative source is available in addition to the MediaSource URL object, it will display the familiar AirPlay icon to the video player control. Should the user select AirPlay, it will switch over from MSE to the AirPlay-compatible URL.

The streaming code for this demo is very basic and serves as an example of getting the bytes from the video to the browser.

document.onload = async () => {
  if (!MediaSource.isTypeSupported(mediaType)) {
    return console.log('Media type is not supported.');
  }

  await new Promise((resolve) => {
    source.addEventListener("sourceopen", resolve, { once: true });
    document.body.appendChild(video);
  });

  const sourceBuffer = source.addSourceBuffer(mediaType);

  async function loadSegment() {
    const response = await fetch(videoURL);
    const buffer = await response.arrayBuffer();
    await new Promise((resolve) => {
      sourceBuffer.addEventListener("updateend", resolve, { once: true });
      sourceBuffer.appendBuffer(buffer);
    });
  }

  if (typeof(source.onstartstreaming) !== 'undefined') {
    source.onstartstreaming = async () => {
      loadSegment();
    };
  } else loadSegment();
});

Feedback

Offering support for MMS/MSE and AirPlay is gives users the best of all worlds and the video element makes it easy to offer multiple sources. You can learn more about the Managed Media Source API proposal at the W3C, and learn about HTTP Live Streaming from Apple’s documentation.

We love to hearing from you. Send a tweet to @webkit to share your thoughts on this feature. Find us on Mastodon at @jensimmons@front-end.social and @jondavis@mastodon.social. You can also follow WebKit on LinkedIn. If you run into any issues, we welcome your WebKit bug reports on WebKit-powered features like this. Reporting issues and sharing your feedback makes an enormous difference.