Currently when there is a failure on clean tree (ToT), API test build summary shows: "Found x pre-existing API test failure". This is fine, but not very informative. It would be better to include the names of the pre-existing failures in the build summary string. Those failing test names would show up on the builder page, and it would be very convenient to notice the test failure pattern over multiple builds.
Created attachment 373522 [details] Patch
Sample runs: https://ews-build.webkit.org/#/builders/9/builds/4064 https://ews-build.webkit.org/#/builders/3/builds/4179 Also notice the build statuses on builder page: https://ews-build.webkit.org/#/builders/3
Comment on attachment 373522 [details] Patch View in context: https://bugs.webkit.org/attachment.cgi?id=373522&action=review > Tools/BuildSlaveSupport/ews-build/steps.py:1088 > + clean_tree_failures_string = ', '.join([failure_name.replace('TestWebKitAPI.', '') for failure_name in clean_tree_failures]) What we call 'API' tests is actually 2 different binaries on Mac, 3 for Windows (at the moment, at least). That first key in the test name is the binary. It could be TestWebKitAPI, TestWTF, TestWTF.exe, TestWebCore.exe or TestWebKitLegacy.exe. I'm not sure it makes sense to strip the binary name at all, run-webkit-tests is smart enough to understand it. If we do want to string the binary name, though, we need to do it with split('.').
Created attachment 382483 [details] Patch
Not removing 'TestWebKitAPI.' string anymore. Also updated the patch to also limit the number of failures to display to 10 (although it should be rare to have more than 10 failures on trunk). This change would not affect the status-bubble tooltip. Sample runs: https://ews-build.webkit-uat.org/#/builders/4/builds/67 https://ews-build.webkit-uat.org/#/builders/4/builds/66
Comment on attachment 382483 [details] Patch Clearing flags on attachment: 382483 Committed r251869: <https://trac.webkit.org/changeset/251869>
All reviewed patches have been landed. Closing bug.
<rdar://problem/56791844>