Announcing SunSpider 0.9.1
Today we’d like to announce SunSpider 0.9.1, a new version of the popular SunSpider JavaScript benchmark that improves accuracy of results. We recommend that anyone using SunSpider to test JavaScript performance should migrate to the new version.
A little over two years ago, we announced the SunSpider JavaScript benchmark. We originally created SunSpider for our own use, to help us speed up WebKit’s JavaScript implementation. Since then, SunSpider has become a widely respected metric of JavaScript performance, and has been cited frequently in performance comparisons. But we’ve had some suggestions for SunSpider that could make it more accurate on fast systems and cross-browser comparisons.
We were hesitant to change the SunSpider content or harness much at all, since it’s been used for cross-version and cross-browser comparisons for so long. But it seems important to address the most significant issues. With that in mind, we have made a limited number of changes to the test harness and to the test content itself.
Dealing with Power Management
Computers and JavaScript implementations have gotten much faster over the past two years. If you look at the comments on the original SunSpider announcement, you can see that most people were reporting results around 10,000ms. Now it’s common for high performance JS engines on fast hardware to get scores around 300ms.
Along with this speedup, we ran into an unanticipated problem with the test harness. By design, the SunSpider harness has significant pauses between each test, to give browsers a chance to settle down after the load. But with fast browsers on modern hardware, this gives enough time for power management to kick in and ratchet down the CPU clock rate. As a result, most of the benchmark does not run at full speed. This issue, originally reported by Mike Belshe of the Chrome team, has more impact on faster JavaScript engines.
This resulted in our first SunSpider harness bug fix: bug 32505: In-browser SunSpider suffers excessive penalty under power management. In addition to lowering the delay between tests, we also greatly reduced the amount of loading that happens during the benchmark. All of the content is in one page. This further reduces the variability between test runs. And as an added bonus, the SunSpider test suite is now much faster to complete, as it no longer waits half a second between every single test.
Test Content Bug Fixes
SunSpider’s test content has some oddities. Out of the various problems reported, two stood out as affecting the accuracy of the test to some extent. First, the string-base64 test had a bug that made it incorrect in Internet Explorer. The test has been updated, and should now give more accurate results in IE. Second, the regexp-dna test had an issue specific to Gecko-based browsers such as Firefox. It too has been updated and should now be more accurate in Firefox.
Versioned Test Content
Since we changed the test content, we thought that some might be interested in running the “classic” SunSpider 0.9 content set under the new harness. We’d also like to be prepared to handle multiple versions of the test content in general, so we did just that. For this release, we made sunspider-0.9.1 the default content set (both command-line and hosted).
Minor Test Harness Tweaks
In the course of implementing these fixes, we decided some cleanup was in order. First, we made sure all of the HTML files in the browser-hosted version are valid HTML5. With HTML5 advancing and becoming more popular, it seemed like the right thing to do. The test will still work fine in older browsers. In addition, the older SunSpider harness processed JSON strings using somewhat ad-hoc techniques. We updated to use json2.js or browser-native JSON.parse, where available.
Try it Out
Last but not least, try it out, report your scores, any problems you run into, or any other interesting comments. You can run it with the latest content set right from that page, or visit the all versions page if you want to test an older content set.