Creating Web Inspector Audits

This post is a followup to the Audits in Web Inspector post, and explains the capabilities of and how to write an audit.

Test Case Format

The actual test that is run in the page is just a stringified JavaScript function. async functions are allowed, as well as non-async functions that return a Promise. The only requirement is that the return value of the function (or Promise) conforms to the following rules:

• Returning true/false will translate to a Pass/Fail result with no additional data.
• Returning a string value will translate to the corresponding result (e.g. "pass" is a Pass result) with no additional data.
• Returning an object gives the most flexibility, as it allows for additional data other than the audit’s result to be displayed in the Audit tab. Result levels are first retrieved from the "level" value (if it exists), and are translated in the same way as if a string was returned (e.g. "pass" is a Pass result). Alternatively, using any result string as a key with a value of true will have the same effect (e.g. "pass": true).

There are five result levels:

• "pass" corresponds to a Pass result, which is where everything was as it should be.
• "warning" corresponds to a Warning result, which is a “soft pass” in that there was nothing wrong, but there were things that should be changed.
• "fail" corresponds to a Fail result, which is an indication that something is not as it should be.
• "error" corresponds to an Error result, which occurs when the JavaScript being run threw an error.
• "unsupported" corresponds to an Unsupported result, which is a special case that can be used to indicate when the data being tested isn’t supported by the current page (e.g. missing some API).

There are also three additional pieces of data that can be returned within the result object and have a dedicated specialized interface:

• domNodes, which is an array of DOM nodes that will be displayed in the Audit tab much the same as if they were logged to the console.
• domAttributes, which is an array of strings, each of which will be highlighted if present on any DOM nodes within domNodes.
• errors, which is an array of Error objects and can be used as a way of exposing errors encountered while running the audit.
  • If this array has any values, the result of the audit is automatically changed to Error.
  • If an error is thrown while running an audit, it will automatically be added to this list.

For custom data, you can add it to the result object and it will display in the Audit tab, so long as it’s JSON serializable and doesn’t overlap with any of the items above.

Container Structure

Web Inspector Audits follow a simple and highly flexible structure in the form of JSON objects. These objects fall into two main categories: tests and groups.

The format for a test is as follows:

{
    "type": "test-case",
    "name": "...",
    "test": "<stringified JavaScript function>"
}

The format for a group is as follows:

{
    "type": "test-group",
    "name": "...",
    "tests": [...]
}

In this case, the values inside tests can be both individual test cases, or additional groups.

Both tests and groups also support a number of optional properties:

• description is a basic string value that is displayed in the Audit tab as a way of providing more information about that specific audit, such as what it does or what it’s trying to test.
• supports can be used as an alternative to feature-checking, in that it prevents the audit from even being run unless the number value matches Web Inspector’s Audit version number, which can be found at the bottom of the Web Inspector window when in edit mode in the Audit tab. At the time of writing this post, the current version is 3.
• setup is similar to a test case’s test value, except that it only has an effect when supplied for a top-level audit. The idea behind this value is to be able to share code between all audits in a group, as it is executed before the first audit in a group.

Specially Exposed Data

Since audits are run from Web Inspector, it’s possible for additional information to be exposed to each test function being executed. Much of this data is already exposed to Web Inspector, but was never accessible via JavaScript in any way.

The information is exposed via a WebInspectorAudit object that is passed to each test function. Note that this object is shared between all test under a given top-level audit, which is defined as an audit with no parent. As such, attaching data to this object to be shared between test is accepted and encouraged.

Version

Accessing Web Inspector’s Audit version number from within a test is as simple as getting the value of WebInspectorAudit.Version.

Resources

The following all relate to dealing with resources loaded by the page, and are held by WebInspectorAudit.Resources.

• getResources() will return a list of objects, each corresponding to a specific resource loaded by the inspected page, identified by a string url, string mimeType, and audit-specific id.
• getResourceContent(id) will return an object with the string data and boolean base64Encoded contents of the resource with the given audit-specific id.

DOM

The following all relate to dealing with the DOM tree, and are held by WebInspectorAudit.Resources.

• hasEventListeners(node[, type]) returns true/false depending on whether the given DOM node has any event listeners, or has an event listener for the given type (if specified).

Accessibility

The following all relate to dealing with the accessibility tree, and are held by WebInspectorAudit.Accessibility. Further information can be found in the WAI-ARIA specification.

• getElementsByComputedRole(role[, container]) returns an array of DOM nodes that match the given role that are children of the container DOM node (if specified) or the main document.
• getActiveDescendant(node) returns the active descendant of the given DOM node.
• getMouseEventNode(node) returns the DOM node that would handle mouse events which is or is a child of the given DOM node.
• getParentNode(node) returns the parent DOM node of the given DOM node in the accessibility tree.
• getChildNodes(node) returns an array of DOM nodes that are children of the given DOM node in the accessibility tree.
• getSelectedChildNodes(node) returns an array of currently selected DOM nodes that are children of the given DOM node in the accessibility tree.
• getControlledNodes(node) returns an array of DOM nodes that are controlled by the given DOM node.
• getFlowedNodes(node) returns an array of DOM nodes that are flowed to from the given DOM node.
• getOwnedNodes(node) returns an array of DOM nodes that are owned by the given DOM node.
• getComputedProperties(node) returns an object that contains various accessibility properties for the given DOM node. Since HTML allows for “incorrect” values in markup (e.g. an invalid value for an attribute), the following properties use the computed value determined by WebKit:
  • busy is a boolean related to the aria-busy attribute.
  • checked is a string related to the aria-checked attribute.
  • currentState is a string related to the aria-current attribute.
  • disabled is a boolean related to the aria-disabled attribute.
  • expanded is a boolean related to the aria-expanded attribute.
  • focused is a boolean that indicates whether the given node is focused.
  • headingLevel is a number related to the aria-level attribute and the various HTML heading elements.
  • hidden is a boolean related to the aria-hidden attribute.
  • hierarchicalLevel is a number related to the aria-level attribute.
  • ignored is a boolean that indicates whether the given node is currently being ignored in the accessibility tree.
  • ignoredByDefault is a boolean that indicates whether the given node is always ignored by the accessibility tree.
  • invaludStatus is a string related to the aria-invalid attribute.
  • isPopUpButton is a boolean related to the aria-haspopup attribute.
  • label is a string related to the aria-label attribute
  • liveRegionAtomic is a boolean related to the aria-atomic attribute.
  • liveRegionRelevant is an array of strings related to the aria-relevant attribute.
  • liveRegionStatus is a string related to the aria-live attribute.
  • pressed is a boolean related to the aria-pressed attribute.
  • readonly is a boolean related to the aria-readonly attribute.
  • required is a boolean related to the aria-required attribute.
  • role is a string related to the role attribute.
  • selected is a boolean related to the aria-selected attribute.

Getting Started

As mentioned in the Audits in Web Inspector post, the Demo Audit and Accessibility audits that are included as part of Web Inspector can be used as good starter templates for the formatting and structure of a Web Inspector audit.

Alternatively, the eslint.json audit, which runs ESLint against every *.js resource that was loaded from the main origin of the inspected page, can serve as a more complex example of an async audit that makes use of some of the above specially exposed data.

Feedback

The Audit tab was added in Safari Technology Preview 75. Is there a workflow that isn’t supported, or a piece of data that isn’t exposed, that you’d like to use when writing/running audits? Do you have an idea for a default audit that should be included in Web Inspector? Let us know! Please feel free to send us feedback on Twitter (@dcrousso or @jonathandavis) or by filing a bug.