Bug 190401 - [iOS Debug] Assertion failure in NSComparisonResult UIContentSizeCategoryCompareToCategory()
Summary: [iOS Debug] Assertion failure in NSComparisonResult UIContentSizeCategoryComp...
Status: RESOLVED FIXED
Alias: None
Product: WebKit
Classification: Unclassified
Component: Tools / Tests (show other bugs)
Version: Other
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: Wenson Hsieh
URL:
Keywords: InRadar
Depends on:
Blocks:
 
Reported: 2018-10-09 09:55 PDT by Dawei Fenton (:realdawei)
Modified: 2018-11-19 13:37 PST (History)
14 users (show)

See Also:


Attachments
Crash log (82.40 KB, text/plain)
2018-10-09 10:43 PDT, Ryan Haddad
no flags Details
Patch (4.04 KB, patch)
2018-11-18 20:12 PST, Wenson Hsieh
no flags Details | Formatted Diff | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Dawei Fenton (:realdawei) 2018-10-09 09:55:46 PDT
API test TestWebKitAPI.EditorStateTests.TypingAttributesTextAlignmentAbsoluteAlignmentOptions is crashing on iOS 12 [Debug]

It crashes ~50 percent of the time.

Sample output:
https://build.webkit.org/builders/Apple%20iOS%2012%20Simulator%20Debug%20WK2%20%28Tests%29/builds/221/steps/run-api-tests/logs/stdio

    TestWebKitAPI.EditorStateTests.TypingAttributesTextAlignmentAbsoluteAlignmentOptions
        2018-10-08 10:01:47.818 TestWebKitAPI[14259:247411] *** Assertion failure in NSComparisonResult UIContentSizeCategoryCompareToCategory(__strong UIContentSizeCategory _Nonnull, __strong UIContentSizeCategory _Nonnull)(), /BuildRoot/Library/Caches/com.apple.xbs/Sources/UIKitCore_Sim/UIKit-3698.84.15/UIContentSizeCategory.m:87
        Child process terminated with signal 11: Segmentation fault
Comment 1 Ryan Haddad 2018-10-09 10:43:28 PDT
Created attachment 351892 [details]
Crash log
Comment 2 Ryan Haddad 2018-10-09 10:44:37 PDT
Dawei, is this reproducible locally?
Comment 3 Dawei Fenton (:realdawei) 2018-10-09 11:05:02 PDT
(In reply to Ryan Haddad from comment #2)
> Dawei, is this reproducible locally?

Attempting this now
Comment 4 Dawei Fenton (:realdawei) 2018-10-09 14:55:06 PDT
This is reproducible with the following command:

While true; do run-api-tests --debug --ios-simulator TestWebKitAPI.EditorStateTests.TypingAttributesTextAlignmentAbsoluteAlignmentOptions ;done
Comment 5 Wenson Hsieh 2018-10-09 17:35:11 PDT
I suspect that making TestWebKitAPI a proper UI application would fix this.
Comment 6 Jonathan Bedard 2018-10-11 09:27:02 PDT
(In reply to Wenson Hsieh from comment #5)
> I suspect that making TestWebKitAPI a proper UI application would fix this.

Making it a UI application would prevent a flakey SEGFAULT? That would strike me as very strange.

Also, are we sure that's the right crashlog? The stack trace isn't pointing to that UIKit assertion error, it's a C++ string/stream error in our code.
Comment 7 Ryan Haddad 2018-10-11 09:30:53 PDT
(In reply to Jonathan Bedard from comment #6)
> (In reply to Wenson Hsieh from comment #5)
> Also, are we sure that's the right crashlog? The stack trace isn't pointing
> to that UIKit assertion error, it's a C++ string/stream error in our code.

That is the one that matched up with the timestamp for the assertion failure in one of the examples I saw. I just found a more recent one and verified that it looks the same.
Comment 8 Radar WebKit Bug Importer 2018-10-11 09:33:28 PDT
<rdar://problem/45195285>
Comment 9 Wenson Hsieh 2018-10-11 10:10:59 PDT
(In reply to Jonathan Bedard from comment #6)
> (In reply to Wenson Hsieh from comment #5)
> > I suspect that making TestWebKitAPI a proper UI application would fix this.
> 
> Making it a UI application would prevent a flakey SEGFAULT? That would
> strike me as very strange.

Yes, I strongly suspect this is the case. We're hitting this exception because UIKit can't figure out what scale to apply to an icon image for a button in the callout bar when focusing an editable element.

UIKit attempts to determine this scale by inspecting -[UIApplication preferredContentSizeCategory], which in the case of these API tests, returns `nil`. However, in WebKitTestRunner, this returns a non-null string, so we don't hit this exception. So I think we either need to make TestWebKitAPI a UI application to address this, or pretend to make it an app by swizzling some (carefully chosen) SPI.

> 
> Also, are we sure that's the right crashlog? The stack trace isn't pointing
> to that UIKit assertion error, it's a C++ string/stream error in our code.

The crash log does not seem relevant to the exception here; I've pasted an example trace in the radar.
Comment 10 Alexey Proskuryakov 2018-10-11 10:20:04 PDT
This may imply that the test itself is flaky, as it doesn't always hit the code path that asserts.
Comment 11 Wenson Hsieh 2018-10-11 10:22:06 PDT
(In reply to Alexey Proskuryakov from comment #10)
> This may imply that the test itself is flaky, as it doesn't always hit the
> code path that asserts.

I think this just means that the test sometimes finishes before UIKit decides to show the callout bar.
Comment 12 Ryan Haddad 2018-11-05 17:17:40 PST
Seeing something similar with another test:

    TestWebKitAPI.WKAttachmentTests.CopyAndPasteBetweenWebViews
        2018-11-05 16:33:02.855 TestWebKitAPI[33977:6055524] *** Assertion failure in NSComparisonResult UIContentSizeCategoryCompareToCategory(__strong UIContentSizeCategory _Nonnull, __strong UIContentSizeCategory _Nonnull)(), /BuildRoot/Library/Caches/com.apple.xbs/Sources/UIKitCore_Sim/UIKit-3698.84.15/UIContentSizeCategory.m:87
        Child process terminated with signal 11: Segmentation fault

https://build.webkit.org/builders/Apple%20iOS%2012%20Simulator%20Release%20WK2%20%28Tests%29/builds/759/steps/run-api-tests/logs/stdio
Comment 13 Ryan Haddad 2018-11-14 15:48:23 PST
And another :( 

    TestWebKitAPI.WKWebViewEditActions.PasteAndMatchStyle
        2018-11-14 15:03:50.448 TestWebKitAPI[57358:101711663] *** Assertion failure in NSComparisonResult UIContentSizeCategoryCompareToCategory(__strong UIContentSizeCategory _Nonnull, __strong UIContentSizeCategory _Nonnull)(), /BuildRoot/Library/Caches/com.apple.xbs/Sources/UIKitCore_Sim/UIKit-3698.84.15/UIContentSizeCategory.m:87
        Child process terminated with signal 11: Segmentation fault
Comment 14 Ryan Haddad 2018-11-14 15:49:19 PST
This is making it difficult to spot legitimate regressions, as there is almost always at least one test hitting this issue.
Comment 15 Wenson Hsieh 2018-11-14 15:54:56 PST
(In reply to Ryan Haddad from comment #14)
> This is making it difficult to spot legitimate regressions, as there is
> almost always at least one test hitting this issue.

I think that ideally, this should be fixed by making TestWebKitAPI a real UI application.

But perhaps it's worth checking in a trivial workaround for this crash in the meantime, so that API test results aren't muddled by these assertions. We could then remove the workaround after the former is done. Does that sound reasonable?
Comment 16 Ryan Haddad 2018-11-14 15:59:06 PST
(In reply to Wenson Hsieh from comment #15)
> (In reply to Ryan Haddad from comment #14)
> > This is making it difficult to spot legitimate regressions, as there is
> > almost always at least one test hitting this issue.
> 
> I think that ideally, this should be fixed by making TestWebKitAPI a real UI
> application.
> 
> But perhaps it's worth checking in a trivial workaround for this crash in
> the meantime, so that API test results aren't muddled by these assertions.
> We could then remove the workaround after the former is done. Does that
> sound reasonable?
Since I don't think moving TestWebKitAPI to a UI application is happening soon (someone correct me if I am wrong), I think it is reasonable to add a workaround.
Comment 17 Jonathan Bedard 2018-11-15 13:15:57 PST
(In reply to Ryan Haddad from comment #16)
> ...
> Since I don't think moving TestWebKitAPI to a UI application is happening
> soon (someone correct me if I am wrong), I think it is reasonable to add a
> workaround.

Unless someone other than me is moving it to a UI application, it will be May/June 2019.
Comment 18 Wenson Hsieh 2018-11-18 20:12:50 PST
Created attachment 355248 [details]
Patch
Comment 19 WebKit Commit Bot 2018-11-19 13:37:09 PST
Comment on attachment 355248 [details]
Patch

Clearing flags on attachment: 355248

Committed r238380: <https://trac.webkit.org/changeset/238380>
Comment 20 WebKit Commit Bot 2018-11-19 13:37:11 PST
All reviewed patches have been landed.  Closing bug.