MotionMark 1.1
Two years ago, we released MotionMark, a benchmark for Web graphics. It runs a series of tests that exercise common ways of painting on the Web. We’ve used this benchmark to improve the performance of WebKit and underlying frameworks, and we’re happy to see that others have used the benchmark for their own performance investigations.
Today we are announcing an update to the MotionMark benchmark, which we’re calling version 1.1.
Test updates
First, let’s do a quick review about the benchmark. MotionMark runs a suite of tests that measures a browser’s painting performance by exercising various techniques, including CSS, text, canvas, and SVG. Each test packages a small set of graphics primitives into a single rendering element, and renders as many of them as possible while maintaining 60 fps. That number represents the test’s score. For more details, you can read our introductory blog post.
We want to expose MotionMark to as many aspects of the rendering engine as possible. In our first update to the benchmark, we’ve refreshed two tests in the suite.
The presentation of the Multiply test, which utilizes DOM element painting, caps the maximum number of elements that can be drawn. MotionMark requires tests to be able to stress browsers such that they cannot maintain 60 fps, and some of Apple’s new hardware, like the iPhone Xs, were powerful enough to animate Multiply at the maximum complexity at 60 fps. So we had to make the test harder! We reduced the physical size of each element to fit more of them on the available space and increase that cap threefold. We also updated the ways that an element can be hidden during the test. Before the elements toggled their visibility using visibility: hidden; now they also do so via display: none and opacity: 0.
In the original 1.0 suite, each rendering element in the Leaves test was an image to which translate and rotate transforms were applied. For 1.1 we use the width and height attributes to paint the images in a range of different sizes, and test blending by animating a fade using opacity.
Leveraging MotionMark’s design for easier analysis
MotionMark has enabled us to monitor our graphics performance in WebKit and underlying system frameworks we depend on like CoreAnimation, as well as find opportunities to progress performance. As our experience in regression analysis with this benchmark grew, we found a couple of processes that facilitated our efforts to isolate bottlenecks and verify performance gains.
First, we split tests from the “main” suite into derivative versions based on specific techniques. For example, with the Multiply test update mentioned earlier, we created versions where all of the elements used only one of the CSS techniques for hiding an element. For the Design test we created a version that only contained Latin text, one that used East Asian characters, and one that used right-to-left and complex scripts. Isolating the techniques made it easier to determine which part of the test WebKit regressed, and magnified problems in traces for easier analysis.
Second, we added a way to run a test more simply, with a fixed number of elements. The benchmark uses a ramping controller that varies the painting load such that it repeatedly crosses over the threshold where the browser can no longer render the scene at 60 fps. Analyzing Instruments traces for such a run can be difficult to track a regression, however, since the load is dynamic from frame to frame. MotionMark’s design allows us to plug in different kinds of controllers, and using one that simply rendered a scene at a static complexity facilitated A/B analyses across versions of WebKit and the OS. Subsequent fixes could then be easily verified using both the fixed and ramping controllers.
Results
Using MotionMark as a tool for tracking regressions and discovering ways to progress have been critical for us to improve graphics performance in WebKit. As a result, we’ve made Safari on macOS Mojave and iOS 12 on average over 20% faster on MotionMark 1.1 than in the previous release.
In conclusion, MotionMark has been updated to version 1.1 to scale better on faster hardware and include more techniques for painting. We hope this benchmark continues to be a useful tool for browsers to optimize their engines for the modern Web.