Import W3C DOM test suite from github.com/w3c/web-platform-tests
Created attachment 260098 [details] Patch
Comment on attachment 260098 [details] Patch Attachment 260098 [details] did not pass mac-ews (mac): Output: http://webkit-queues.webkit.org/results/114504 New failing tests: http/tests/w3c/dom/nodes/Document-createElement-namespace-tests/bare_mathml.svg http/tests/w3c/dom/nodes/Document-createElement-namespace-tests/minimal_html.svg http/tests/w3c/dom/nodes/Document-createElement-namespace-tests/bare_xhtml.svg http/tests/w3c/dom/nodes/Document-createElement-namespace.html http/tests/w3c/dom/nodes/Document-createElement-namespace-tests/bare_svg.svg http/tests/w3c/dom/nodes/Document-createElement-namespace-tests/xhtml_ns_removed.svg http/tests/w3c/dom/nodes/Document-createElement-namespace-tests/xhtml.svg http/tests/w3c/dom/nodes/Document-createElement-namespace-tests/svg.svg http/tests/w3c/dom/nodes/Document-createElement-namespace-tests/xhtml_ns_changed.svg http/tests/w3c/dom/nodes/Document-createElement-namespace-tests/mathml.svg
Created attachment 260104 [details] Archive of layout-test-results from ews101 for mac-mavericks The attached test failures were seen while running run-webkit-tests on the mac-ews. Bot: ews101 Port: mac-mavericks Platform: Mac OS X 10.9.5
Created attachment 260106 [details] Archive of layout-test-results from ews107 for mac-mavericks-wk2 The attached test failures were seen while running run-webkit-tests on the mac-wk2-ews. Bot: ews107 Port: mac-mavericks-wk2 Platform: Mac OS X 10.9.5
Created attachment 260111 [details] Patch
Comment on attachment 260111 [details] Patch Attachment 260111 [details] did not pass mac-ews (mac): Output: http://webkit-queues.webkit.org/results/114975 New failing tests: http/tests/w3c/dom/nodes/Document-createElement-namespace.html
Created attachment 260113 [details] Archive of layout-test-results from ews100 for mac-mavericks The attached test failures were seen while running run-webkit-tests on the mac-ews. Bot: ews100 Port: mac-mavericks Platform: Mac OS X 10.9.5
Created attachment 260114 [details] Archive of layout-test-results from ews106 for mac-mavericks-wk2 The attached test failures were seen while running run-webkit-tests on the mac-wk2-ews. Bot: ews106 Port: mac-mavericks-wk2 Platform: Mac OS X 10.9.5
Created attachment 260115 [details] Patch
http://www.w3.org/Consortium/Legal/2008/04-testsuite-copyright.html says that this is available under three-clause BSD license, which is good. Can't really review an 8-megabyte patch. What are the tests that need http there? Can others be run without http?
Also, how long do these tests take to run?
(In reply to comment #10) > http://www.w3.org/Consortium/Legal/2008/04-testsuite-copyright.html says > that this is available under three-clause BSD license, which is good. > > Can't really review an 8-megabyte patch. What are the tests that need http > there? Can others be run without http? Yes, some of them do require HTTP, which is why I moved it there. I did not want to split the tests because it will make updating (re-importing) those tests more difficult.
(In reply to comment #11) > Also, how long do these tests take to run? Those tests are pretty fast, it takes less than 30 seconds locally. There ~250 tests if I remember correctly.
Is it 30 seconds in debug build, or in release, and is the machine a fast one? If it adds 30 seconds to a release test run on a 12-core Mac, then we may need to look into making these faster.
Created attachment 260228 [details] Patch
(In reply to comment #14) > Is it 30 seconds in debug build, or in release, and is the machine a fast > one? If it adds 30 seconds to a release test run on a 12-core Mac, then we > may need to look into making these faster. I have skipped a few slow tests in debug. I have measured that the run time is now ~30 seconds for a debug build on a MacBookPro, including the http server bring up / tear down. Hopefully, this is acceptable? This brings ~200 tests with good coverage of the DOM. Also note that there are a lot of checks that we are currently failing and which I started to work on.
Comment on attachment 260228 [details] Patch rs=me
Comment on attachment 260228 [details] Patch Clearing flags on attachment: 260228 Committed r189155: <http://trac.webkit.org/changeset/189155>
All reviewed patches have been landed. Closing bug.
One failure on bots: http/tests/w3c/dom/nodes/Element-matches.html @@ -171,7 +171,7 @@ PASS In-document Element.matches: :empty pseudo-class selector, matching all empty elements (with no refNodes): #pseudo-empty :empty PASS In-document Element.matches: :link and :visited pseudo-class selectors, matching a and area elements with href attributes (with no refNodes): #pseudo-link :link, #pseudo-link :visited PASS In-document Element.matches: :link and :visited pseudo-class selectors, matching link elements with href attributes (with no refNodes): #head :link, #head :visited -PASS In-document Element.matches: :target pseudo-class selector, matching the element referenced by the URL fragment identifier (with no refNodes): :target +FAIL In-document Element.matches: :target pseudo-class selector, matching the element referenced by the URL fragment identifier (with no refNodes): :target assert_true: The element #target should match the selector. expected true got false PASS In-document Element.matches: :lang pseudo-class selector, matching inherited language (with no refNodes): #pseudo-lang-div1:lang(en) PASS In-document Element.matches: :lang pseudo-class selector, matching specified language with exact value (with no refNodes): #pseudo-lang-div2:lang(fr) PASS In-document Element.matches: :lang pseudo-class selector, matching specified language with partial value (with no refNodes): #pseudo-lang-div3:lang(en) EWS missed it because it treats flaky tests as passing.
(In reply to comment #20) > One failure on bots: > > http/tests/w3c/dom/nodes/Element-matches.html > > @@ -171,7 +171,7 @@ > PASS In-document Element.matches: :empty pseudo-class selector, matching > all empty elements (with no refNodes): #pseudo-empty :empty > PASS In-document Element.matches: :link and :visited pseudo-class > selectors, matching a and area elements with href attributes (with no > refNodes): #pseudo-link :link, #pseudo-link :visited > PASS In-document Element.matches: :link and :visited pseudo-class > selectors, matching link elements with href attributes (with no refNodes): > #head :link, #head :visited > -PASS In-document Element.matches: :target pseudo-class selector, matching > the element referenced by the URL fragment identifier (with no refNodes): > :target > +FAIL In-document Element.matches: :target pseudo-class selector, matching > the element referenced by the URL fragment identifier (with no refNodes): > :target assert_true: The element #target should match the selector. expected > true got false > PASS In-document Element.matches: :lang pseudo-class selector, matching > inherited language (with no refNodes): #pseudo-lang-div1:lang(en) > PASS In-document Element.matches: :lang pseudo-class selector, matching > specified language with exact value (with no refNodes): > #pseudo-lang-div2:lang(fr) > PASS In-document Element.matches: :lang pseudo-class selector, matching > specified language with partial value (with no refNodes): > #pseudo-lang-div3:lang(en) > > EWS missed it because it treats flaky tests as passing. Oh, I simply rebaselined it in https://trac.webkit.org/r189158 when I saw it failed on all bots. If you're saying it's flaky, I guess I should mark this test as flaky instead?
It seems more likely that it's not randomly flaky, but that it fails when run after a certain other test - meaning that there is a real WebCore bug.
(In reply to comment #22) > It seems more likely that it's not randomly flaky, but that it fails when > run after a certain other test - meaning that there is a real WebCore bug. Agreed, I will investigate.
There are Windows test failures too: http/tests/w3c/dom/interfaces.html http/tests/w3c/dom/traversal/NodeFilter-constants.html https://build.webkit.org/results/Apple%20Win%207%20Release%20(Tests)/r189161%20(53771)/results.html
(In reply to comment #24) > There are Windows test failures too: > > http/tests/w3c/dom/interfaces.html > http/tests/w3c/dom/traversal/NodeFilter-constants.html > > https://build.webkit.org/results/Apple%20Win%207%20Release%20(Tests)/ > r189161%20(53771)/results.html I am working on at least the second one via https://bugs.webkit.org/show_bug.cgi?id=148602. This is an actual bug in the implementation :) I'll check out the second one.
rdar://problem/22543901
(In reply to comment #12) > (In reply to comment #10) > > http://www.w3.org/Consortium/Legal/2008/04-testsuite-copyright.html says > > that this is available under three-clause BSD license, which is good. > > > > Can't really review an 8-megabyte patch. What are the tests that need http > > there? Can others be run without http? > > Yes, some of them do require HTTP, which is why I moved it there. I did not > want to split the tests because it will make updating (re-importing) those > tests more difficult. https://bugs.webkit.org/show_bug.cgi?id=127095#c11 has some interesting information about how Firefox is doing. Re-importing is important and should be made as easy as possible. In general, I think all wpt tests should go to imported/w3c/web-platform-tests. If at some point, we see that tests take too long to run due to being served by HTTP, we might want to evaluate how to improve things. Also, we should try to import tests from the same github revision, hence using the import-w3c-tests script. I will try to take some time to resync the different tests with a recent wpt revision (see bug 149645 and bug 149656).