We are announcing an update to the Speedometer benchmark. This is a minor version bump fixing the benchmark harness to improve the accuracy and stability of how we measure the speed of the rendering engine.
setTimeout nesting level and throttling
Speedometer was designed to include asynchronous work in its work time measurement when computing a score. This is needed because some browser engines use an optimization strategy of deferring some work to reduce the run time of synchronous operations. To measure the time used for asynchronous work, the Speedometer test harness was using
setTimeout to compute the end time after the asynchronous work is done.
setTimeout inside the test harness for each subtest piles up the
setTimeout nesting level. According to the specification, this triggers the insertion of an artificial 4ms throttle between each
setTimeout callback for energy efficiency, and results in the measured work time including this throttling. This is not ideal because (1) we reach a deep nesting level due to how the test harness is constructed, not due to test content, and (2) the intended purpose of using
setTimeout in the harness is to measure the time of deferred asynchronous work, not artificial throttling.
This update inserts
window.requestAnimationFrame for each test run. This resets the
setTimeout nesting level added by the test harness and other subtests, and increases the accuracy and stability of the work time measurement by isolating each subtest more. We observed a 3-5% improvement in stable Safari, Chrome, and Firefox.
Speedometer 2.1 improves the accuracy and stability of the score measurement by ensuring that it captures the work time of asynchronous work without inducing web engine timer throttling.