Bug 196905

Summary: API test WKAttachmentTests.AddAttachmentToConnectedImageElement is a flaky failure on Mac Release builds
Product: WebKit Reporter: Andy Estes <aestes>
Component: New BugsAssignee: Wenson Hsieh <wenson_hsieh>
Status: RESOLVED FIXED    
Severity: Normal CC: aakash_jain, aestes, bdakin, commit-queue, rniwa, thorton, webkit-bug-importer, wenson_hsieh
Priority: P2 Keywords: InRadar
Version: WebKit Local Build   
Hardware: Mac   
OS: All   
See Also: https://bugs.webkit.org/show_bug.cgi?id=177484
https://bugs.webkit.org/show_bug.cgi?id=196893
https://bugs.webkit.org/show_bug.cgi?id=196909
Attachments:
Description Flags
Patch
none
Add radar link to ChangeLog
none
Also reenable the test none

Andy Estes
Reported 2019-04-14 08:37:51 PDT
On macOS Mojave, in a Release build of r244241, the API test WKAttachmentTests.AddAttachmentToConnectedImageElement failed on 6 out of 10 runs of the WKAttachmentTests API test suite. It also sometimes fails on api-mac EWS bots in a way that makes it appear like a seemingly-unrelated patch regressed the test (e.g., bug #177484). Steps to reproduce: 1. Create a Mac Release build of r244241. 2. run-api-tests WKAttachmentTests. Expected results: All 31 tests pass. Actual results: On 6 out of 10 runs, WKAttachmentTests.AddAttachmentToConnectedImageElement failed. TestWebKitAPI.WKAttachmentTests.AddAttachmentToConnectedImageElement _RegisterApplication(), FAILED TO establish the default connection to the WindowServer, _CGSDefaultConnection() is NULL. 2019-04-14 08:23:20.508 TestWebKitAPI[59497:767481] Expected inserted attachments: ( ) to match ( "<_WKAttachment 0x7fbcc7d01340 id='2414efeb-bbf1-4650-83ce-a167b0a11484'>" ). /Volumes/Data/Projects/OpenSource/Tools/TestWebKitAPI/Tests/WebKitCocoa/WKAttachmentTests.mm:150 Value of: insertedAttachmentsMatch Actual: false Expected: true
Attachments
Patch (4.25 KB, patch)
2019-04-14 18:24 PDT, Wenson Hsieh 		no flags
DetailsFormatted DiffDiff
Add radar link to ChangeLog (4.27 KB, patch)
2019-04-14 18:27 PDT, Wenson Hsieh 		no flags
DetailsFormatted DiffDiff
Also reenable the test (4.46 KB, patch)
2019-04-14 19:00 PDT, Wenson Hsieh 		no flags
DetailsFormatted DiffDiff
Show Obsolete (2) View All Add attachment proposed patch, testcase, etc.
Radar WebKit Bug Importer
Comment 1 2019-04-14 09:34:26 PDT
<rdar://problem/49886096>
Aakash Jain
Comment 2 2019-04-14 15:47:37 PDT
This flaky test is slowing down EWS (since EWS has to re-run API tests few times to confirm that the failure is not because of the patch being tested). e.g.: https://ews-build.webkit.org/#/builders/3/builds/535 https://ews-build.webkit.org/#/builders/3/builds/530 https://ews-build.webkit.org/#/builders/3/builds/528 https://ews-build.webkit.org/#/builders/3/builds/503 https://ews-build.webkit.org/#/builders/3/builds/500 https://ews-build.webkit.org/#/builders/3/builds/467 https://ews-build.webkit.org/#/builders/3/builds/465 https://ews-build.webkit.org/#/builders/3/builds/460 https://ews-build.webkit.org/#/builders/3/builds/541 https://ews-build.webkit.org/#/builders/3/builds/527 https://ews-build.webkit.org/#/builders/3/builds/526 https://ews-build.webkit.org/#/builders/3/builds/518 https://ews-build.webkit.org/#/builders/3/builds/515 https://ews-build.webkit.org/#/builders/3/builds/509 https://ews-build.webkit.org/#/builders/3/builds/499 https://ews-build.webkit.org/#/builders/3/builds/497 https://ews-build.webkit.org/#/builders/3/builds/494 https://ews-build.webkit.org/#/builders/3/builds/493 https://ews-build.webkit.org/#/builders/3/builds/490 https://ews-build.webkit.org/#/builders/3/builds/480 https://ews-build.webkit.org/#/builders/3/builds/479 https://ews-build.webkit.org/#/builders/3/builds/477 https://ews-build.webkit.org/#/builders/3/builds/476 https://ews-build.webkit.org/#/builders/3/builds/474 https://ews-build.webkit.org/#/builders/3/builds/473 https://ews-build.webkit.org/#/builders/3/builds/470 In following builds, the test was so flaky that EWS incorrectly thought that the failure was introduced by the patch being tested (this test failed twice with the patch, and didn't fail on clean tree): https://ews-build.webkit.org/#/builders/3/builds/542 https://ews-build.webkit.org/#/builders/3/builds/546 https://ews-build.webkit.org/#/builders/3/builds/512 https://ews-build.webkit.org/#/builders/3/builds/495 https://ews-build.webkit.org/#/builders/3/builds/492 https://ews-build.webkit.org/#/builders/3/builds/488 https://ews-build.webkit.org/#/builders/3/builds/484 https://ews-build.webkit.org/#/builders/3/builds/471 https://ews-build.webkit.org/#/builders/3/builds/468 https://ews-build.webkit.org/#/builders/3/builds/464 https://ews-build.webkit.org/#/builders/3/builds/454 https://ews-build.webkit.org/#/builders/3/builds/440 https://ews-build.webkit.org/#/builders/3/builds/439 https://ews-build.webkit.org/#/builders/3/builds/429 https://ews-build.webkit.org/#/builders/3/builds/426 https://ews-build.webkit.org/#/builders/3/builds/414 Observation: This test seems to be failing/flaky since https://ews-build.webkit.org/#/builders/3/builds/414
Wenson Hsieh
Comment 3 2019-04-14 16:44:26 PDT
I can't seem to hit this when running the test as-is, but this is reasonably easy to reproduce by running it in a loop like so: --- a/Tools/TestWebKitAPI/Tests/WebKitCocoa/WKAttachmentTests.mm +++ b/Tools/TestWebKitAPI/Tests/WebKitCocoa/WKAttachmentTests.mm @@ -1348,6 +1348,8 @@ TEST(WKAttachmentTests, ChangeFileWrapperForPastedImage) TEST(WKAttachmentTests, AddAttachmentToConnectedImageElement) { + for (int i = 0; i < 50; i++) { + NSLog(@"<WK>: Running iteration: %tu", i); auto webView = webViewForTestingAttachments(); [webView _synchronouslyExecuteEditCommand:@"InsertHTML" argument:@"<img></img>"]; @@ -1370,6 +1372,7 @@ TEST(WKAttachmentTests, AddAttachmentToConnectedImageElement) [attachment synchronouslySetFileWrapper:firstImage.get() newContentType:nil error:nil]; [webView waitForImageElementSizeToBecome:CGSizeMake(215, 174)]; + } }
Wenson Hsieh
Comment 4 2019-04-14 18:24:51 PDT Comment hidden (obsolete)
Wenson Hsieh
Comment 5 2019-04-14 18:27:38 PDT
Created attachment 367409 [details] Add radar link to ChangeLog
Wenson Hsieh
Comment 6 2019-04-14 19:00:59 PDT
Created attachment 367410 [details] Also reenable the test
WebKit Commit Bot
Comment 7 2019-04-14 21:47:44 PDT
Comment on attachment 367410 [details] Also reenable the test Clearing flags on attachment: 367410 Committed r244251: <https://trac.webkit.org/changeset/244251>
WebKit Commit Bot
Comment 8 2019-04-14 21:47:46 PDT
All reviewed patches have been landed. Closing bug.
Note You need to log in before you can comment on or make changes to this bug.