Testing Contributions

Web Developers are encouraged to download a WebKit Build Archive to try out new features. If you come across a bug, make sure to report it to the bug tracker.

Get Set Up

  • Make a Bugzilla account at bugs.webkit.org.
  • Add Tools/Scripts to your shell path so scripts are easy to run.

Verify the Problem

Create a Testcase

For step by step instructions see Creating Layout Tests.

  • See the documentation on running tests, writing new tests, and Writing Layout Tests for DumpRenderTree
  • Run run-webkit-tests to see if your bug fix fixes an existing broken testcase. (If it breaks a test that passed before, be sure your fix is correct before submitting it.)
    • If you want to run a subset of tests, pass in (one or more) paths to the specific files or directories. For example:
      run-webkit-tests --debug --no-build LayoutTests/imported/w3c/web-platform-tests
  • Otherwise, create some testcases.
    • In general, contributing a Web Platform Test (e.g. a reftest or testharness.js test) is preferred when possible. These are stored in LayoutTests/imported/w3c/web-platform-tests/ and kept in sync with the upstream WPT repository.
    • Many older tests are HTML files containing JavaScript that exercise a single feature or sub-feature and produce a reliable, easily recognizable result. For example, see fast/events/event-creation.html or fast/dom/HTMLSelectElement/listbox-select-reset.html.
    • It’s also possible to create a testcase that produces a PNG file and checksum to be compared with an expected result, if you’re testing things like colors. For example, see fast/dom/css-rule-functions.html. For that, be sure not to use testRunner.dumpAsText().
  • Create the expected result file(s).
    • If your fix fixes an existing test (or breaks one, but you’re certain it’s correct), delete the existing expected result(s) for that test so a new expected result can be generated.
      • Expectations are stored as a matching -expected.txt or -expected.html file alongside the test.
      • File-level pass/fail expectations are listed in LayoutTests/TestExpectations.
      • Platform-specific expectations are kept in LayoutTests/platform/, for cases where there are platform differences.
    • Run your new test(s). To run only a subset of the full test suite, run run-webkit-tests with the paths of the test directories or files relative to your LayoutTests directory. For example, run-webkit-tests fast/dom/HTMLSelectElement/ will run all the tests in that directory. If you want to generate PNGs and checksum files too, add the -p option to run-webkit-tests.
    • After the tests are done, the test system will launch your newly built Safari and offer to show the actual results of any tests that have no expected results, as well as diffs of the expected and actual results when applicable. It will automatically create the expected result files for any new tests, so look at their results and make sure they’re correct.
    • If you need to make further changes, you can remove an expected result file and re-run its test to generate a new one. Or, you can run the test with the old expected results in place, and copy the new actual result from /tmp to the correct expected result file. (Don’t just copy and paste from the window, because it’s hard to make sure that all the whitespace is exactly right.)
  • When you’re happy with the expected result, run the test again to be sure it now passes.

Prepare the Change

Submit It

  • Use git-webkit pr to create a pull request for your changes.
  • Edit the commit message template to add the bug reference(s) and brief descriptions for each change, following the examples in the template.
  • You will receive a URL for your pull request in the output.
  • If you created or modified some Web Platform Tests, don’t forget to upstream them to WPT.