Announcing MotionMark 1.3

We are glad to announce an update to MotionMark, Apple’s graphics benchmark for web browsers, taking it to version 1.3. This is a minor update, intended to improve the behavior of the benchmark on a wider variety of hardware devices and to improve the reliability of the scores.

The most significant behavior change is that the benchmark now adapts to the browser’s refresh rate, automatically detecting the frequency of requestAnimationFrame callbacks and using the result to set the target frame rate. Previously, it assumed that the target was 60 frames per second; if the browser called requestAnimationFrame at some different frequency, the results would be invalid.

With this change, comparing benchmark scores between browsers on devices with non-60Hz displays requires some care; a browser that chooses to run requestAnimationFrame at 120Hz has less time to do work per frame, so it will return a lower score. We advise that the display refresh rate is set to 60Hz for such comparisons, as described in the About page.

Minor changes were made to various sub-tests for improved reliability. The maximum complexity of the Multiply subtest was increased because newer, faster configurations could easily hit the cap. The way that elements are hidden in Multiply was changed to ensure only the visible elements contribute to the rendering cost. The Paths test was tweaked to ensure a more consistent workload. And finally, the Design subtest was changed to avoid text getting clipped out at the stage boundary, which would lead to artificially high scores.

There are two scoring-related changes. First, scoring previously chose between a “flat” profile and a “slope” profile based on which had the lowest variance, but the “flat” profile often produced inaccurate scores; it was removed. Second, this version changes which particular frames contribute to the score. The benchmark works by ramping the scene complexity and measuring the duration of some number of frames at each complexity level. In the HTML and SVG sub-tests, the changes in scene complexity involve DOM mutations, in the form of removing elements (since complexity ramps down as the test runs). So there are two types of frames: “mutation” frames and “animation” frames. Previously, the durations of both types of frames fed into the score computation. However, “mutation” frames generally take longer because DOM mutation tends to be more expensive than just animating existing elements, and this adds noise to the results. Since MotionMark is intended to be an animation benchmark, it makes sense for only “animation” frames to contribute to the score.

MotionMark development is now in GitHub; we welcome your participation there.

Change log

• Automatic frame rate detection
• Ignore mutation frames when scoring (with preparatory changes 1 and 2)
• Make the “slope” profile the default profile
• Adjust the maximum complexity of the Multiply subtest
• Use ‘display’ style as the default visibility property for the Multiply test
• Fix instability in the Design subtest
• Make the Paths subtest workload more consistent
• The first frame of each ramp has an erroneously long frame duration