WebDriver Support in Safari 10

As web content has become increasingly interactive, responsive, and complicated, ensuring a good user experience across multiple platforms and browsers is a huge challenge for web developers and QA organizations. Not only must web content be loaded, executed, and rendered correctly, but a user must be able to interact with the content as intended, no matter the browser, platform, hardware, screen size, or other conditions. If your shopping cart application has a clean layout and beautiful typography, but the checkout button doesn’t work because of a configuration mishap in production—or an incorrect CSS rule hides the checkout button in landscape mode—then your users are not going to have a great experience.

Ensuring a good user experience via automation is difficult in part because each browser has its own way of interpreting keyboard and mouse inputs, performing navigations, organizing windows and tabs, and so on. Of course, a human could manually test every interaction and workflow of their website on every browser prior to deploying an update to production, but this is slow, costly, and impractical for all but the largest companies. The Selenium open source project was created to address this gap in automation capabilities for browsers by providing a common API for automation. WebDriver is Selenium’s cross-platform, cross-browser automation API. Using WebDriver, a developer can write an automated test that exercises their web content, and run the test against any browser with a WebDriver-compliant driver.

Safari + WebDriver

I am thrilled to announce that Safari now provides native support for the WebDriver API. Starting with Safari 10 on OS X El Capitan and macOS Sierra, Safari comes bundled with a new driver implementation that’s maintained by the Web Developer Experience team at Apple. Safari’s driver is launchable via the /usr/bin/safaridriver executable, and most client libraries provided by Selenium will automatically launch the driver this way without further configuration.

I’ll first introduce WebDriver by example and explain a little bit about how it’s implemented and how to use Safari’s new driver implementation. I will introduce some important safeguards that are unique to Safari’s driver implementation, and discuss how these may affect you when retargeting an existing WebDriver test suite to run using Safari’s driver.

WebDriver By Example

WebDriver is a browser automation API that enables people to write automated tests of their web content using Python, Java, PHP, JavaScript, or any other language with a library that understands the de facto Selenium Wire Protocol or upcoming W3C WebDriver protocol.

Below is a WebDriver test suite for the WebKit feature status page written in Python. The first test case searches the feature status page for “CSS” and ensures that at least one result appears. The second test case counts the number of visible results when various filters are applied to make sure that each result shows up in at most one filter. Note that in the Python WebDriver library each method call synchronously blocks until the operation completes (such as navigating or executing JavaScript), but some client libraries provide a more asynchronous API.

from selenium.webdriver.common.by import By
from selenium import webdriver
import unittest

class WebKitFeatureStatusTest(unittest.TestCase):

    def test_feature_status_page_search(self):
        self.driver.get("https://webkit.org/status/")

        # Enter "CSS" into the search box.
        search_box = self.driver.find_element_by_id("search")
        search_box.send_keys("CSS")
        value = search_box.get_attribute("value")
        self.assertTrue(len(value) > 0)
        search_box.submit()

        # Count the results.
        feature_count = self.shown_feature_count()
        self.assertTrue(len(feature_count) > 0)

    def test_feature_status_page_filters(self):
        self.driver.get("https://webkit.org/status/")

        filters = self.driver.find_element(By.CSS_SELECTOR, "ul#status-filters li input[type=checkbox]")
        self.assertTrue(len(filters) is 7)

        # Make sure every filter is turned off.
        for checked_filter in filter(lambda f: f.is_selected(), filters):
            checked_filter.click()

        # Count up the number of items shown when each filter is checked.
        unfiltered_count = self.shown_feature_count()
        running_count = 0
        for filt in filters:
            filt.click()
            self.assertTrue(filt.is_selected())
            running_count += self.shown_feature_count()
            filt.click()

        self.assertTrue(running_count is unfiltered_count)

    def shown_feature_count(self):
        return len(self.driver.execute_script("return document.querySelectorAll('li.feature:not(.is-hidden)')"))

def setup_module(module):
    WebKitFeatureStatusTest.driver = webdriver.Safari()

def teardown_module(module):
    WebKitFeatureStatusTest.driver.quit()

if __name__ == '__main__':
    unittest.main()

Note that this code example is for illustration purposes only. This test is contemporaneous with a specific version of the Feature Status page and uses page-specific DOM classes and selectors. So, it may or may not work on later versions of the Feature Status page. As with any test, it may need to be modified to work with later versions of the software it tests.

WebDriver Under the Hood

WebDriver is specified in terms of a REST API; Safari’s driver provides its own local web server that accepts REST-style HTTP requests. Under the hood, the Python Selenium library translates each method call on self.driver into a REST API command and sends the corresponding HTTP request to local web server hosted by Safari’s driver. The driver validates the contents of the request and forwards the command to the appropriate browser instance. Typically, the low-level details of performing most of these commands are delegated to WebAutomationSession and related classes in WebKit’s UIProcess and WebProcess layers. When a command has finished executing, the driver finally sends an HTTP response for the REST API command. The Python client library interprets the HTTP response and returns the result back to the test code.

Running the Example in Safari

To run this WebDriver test using Safari, first you need to grab a recent release of the Selenium open source project. Selenium’s Java and Python client libraries offer support for Safari’s native driver implementation starting in the 3.0.0-beta1 release. Note that the Apple-developed driver is unrelated to the legacy SafariDriver mentioned in the Selenium project. The old SafariDriver implementation is no longer maintained and should not be used. You do not need to download anything besides Safari 10 to get the Apple-developed driver.

Once you have obtained, installed, and configured the test to use the correct Selenium library version, you need to configure Safari to allow automation. As a feature intended for developers, Safari’s WebDriver support is turned off by default. To turn on WebDriver support, do the following:

• Ensure that the Develop menu is available. It can be turned on by opening Safari preferences (Safari > Preferences in the menu bar), going to the Advanced tab, and ensuring that the Show Develop menu in menu bar checkbox is checked.
• Enable Remote Automation in the Develop menu. This is toggled via Develop > Allow Remote Automation in the menu bar.
• Authorize safaridriver to launch the webdriverd service which hosts the local web server. To permit this, run /usr/bin/safaridriver once manually and complete the authentication prompt.

Once you have enabled Remote Automation, copy and save the test as test_webkit.py. Then run the following:

python test_webkit.py

Safari-exclusive Safeguards

While it’s awesome to be able to write automated tests that run in Safari, a REST API for browser automation introduces a potential vector for malicious software. To support a valuable automation API without sacrificing a user’s privacy or security, Safari’s driver implementation introduces extra safeguards that no other browser or driver has implemented to date. These safeguards also double as extra insurance against “flaky tests” by providing enhanced test isolation. Some of these safeguards are described below.

When running a WebDriver test in Safari, test execution is confined to special Automation windows that are isolated from normal browsing windows, user settings, and preferences. Automation windows are easy to recognize by their orange Smart Search field. Similar to browsing in a Private Browsing window, WebDriver tests that run in an Automation window always start from a clean slate and cannot access Safari’s normal browsing history, AutoFill data, or other sensitive information. Aside from the obvious privacy benefits, this restriction also helps to ensure that tests are not affected by a previous test session’s persistent state such as local storage or cookies.

As a WebDriver test is executing in an Automation window, any attempts to interact with the window or web content could derail the test by unexpectedly changing the state of the page. To prevent this from happening, Safari installs a “glass pane” over the Automation window while the test is running. This blocks any stray interactions (mouse, keyboard, resizing, and so on) from affecting the Automation window. If a running test gets stuck or fails, it’s possible to interrupt the running test by “breaking” the glass pane. When an automation session is interrupted, the test’s connection to the browser is permanently severed and the automation window remains open until closed manually.

Along with being able to interact with a stopped test, Safari’s driver implementation also allows for the full use of Web Inspector during and after the execution of a WebDriver test. This is very useful when trying to figure out why a test has hung or is providing unexpected results. safaridriver supports two special WebDriver capabilities just for debugging. The automaticInspection capability will preload the Web Inspector and JavaScript debugger in the background; to pause test execution and bring up Web Inspector’s Debugger tab, you can simply evaluate a debugger; statement in the test page. The automaticProfiling capability will preload the Web Inspector and start a timeline recording in the background; if you later want to see details of what happened during the test, you can open the Timelines tab in Web Inspector to see the captured timeline recording in its entirety.

Safari’s driver restricts the number of active WebDriver sessions. Only one Safari browser instance can be running at any given time, and only one WebDriver session can be attached to the browser instance at a time. This ensures that the user behavior being simulated (mouse, keyboard, touch, etc.) closely matches what a user can actually perform with real hardware in the macOS windowing environment. For example, the system’s text insertion caret can only be active in one window and form control at a time; if Safari’s driver were to allow multiple tests to run concurrently in separate windows or browsers, the text insertion behavior (and resulting DOM focus/blur events) would not be representative of what a user could perform themselves.

Summary

With the introduction of native WebDriver support in Safari 10, it’s now possible to run the same automated tests of web content on all major browsers equally, without writing any browser-specific code. I hope WebDriver support will help web developers  catch user-visible regressions in their web content much more quickly and with less manual testing. Safari’s support comes with new, exclusive safeguards to simultaneously protect user security and privacy and also help you write more stable and consistent tests. You can try out Safari’s WebDriver support today by installing a beta of macOS Sierra or Safari 10.

This is our first implementation of WebDriver for Safari. If you encounter unexpected behavior in Safari’s WebDriver support, or have other suggestions for improvement, please file a bug at https://bugreport.apple.com/. For quick questions or feedback, email web-evangelist@apple.com or tweet @webkit or @brrian on Twitter. Happy testing!

Addendum

After this post was published, we received a lot of useful feedback from users. This section contains information on some common problems and how to work around them.

safaridriver quits immediately after launch when running Safari 10 on El Capitan

Some users reported that they could not get safaridriver to run correctly after installing Safari 10 on El Capitan. The main symptom of this issue is that safaridriver will quit immediately when launched manually via the command line. This happens because a launchd plist used by safaridriver is not automatically loaded. Here’s the workaround:

# Check the status of the com.apple.webdriverd launchd plist:
launchctl list | grep webdriverd

# If it's not loaded, use this command to load the launchd plist:
launchctl load /System/Library/LaunchAgents/com.apple.webdriverd.plist

This issue only manifests on El Capitan when installing a newer Safari through an application update instead of an OS update. Users running safaridriver on macOS Sierra should be unaffected.