Speedometer 2.1

We are announcing an update to the Speedometer benchmark. This is a minor version bump fixing the benchmark harness to improve the accuracy and stability of how we measure the speed of the rendering engine.

setTimeout nesting level and throttling

Speedometer was designed to include asynchronous work in its work time measurement when computing a score. This is needed because some browser engines use an optimization strategy of deferring some work to reduce the run time of synchronous operations. To measure the time used for asynchronous work, the Speedometer test harness was using setTimeout to compute the end time after the asynchronous work is done.

However, using setTimeout inside the test harness for each subtest piles up the setTimeout nesting level. According to the specification, this triggers the insertion of an artificial 4ms throttle between each setTimeout callback for energy efficiency, and results in the measured work time including this throttling. This is not ideal because (1) we reach a deep nesting level due to how the test harness is constructed, not due to test content, and (2) the intended purpose of using setTimeout in the harness is to measure the time of deferred asynchronous work, not artificial throttling.

This update inserts window.requestAnimationFrame for each test run. This resets the setTimeout nesting level added by the test harness and other subtests, and increases the accuracy and stability of the work time measurement by isolating each subtest more. We observed a 3-5% improvement in stable Safari, Chrome, and Firefox.

Conclusion

Speedometer 2.1 improves the accuracy and stability of the score measurement by ensuring that it captures the work time of asynchronous work without inducing web engine timer throttling.