204409 2019-11-20 05:14:37 -0800 [EWS] iOS layout-tests fails with 30+ failures on re-run and on clean-tree 2019-11-20 12:35:31 -0800 1 1 1 Unclassified WebKit New Bugs Other Unspecified Unspecified RESOLVED FIXED https://bugs.webkit.org/show_bug.cgi?id=203792 https://bugs.webkit.org/show_bug.cgi?id=204415 https://bugs.webkit.org/show_bug.cgi?id=204416 https://bugs.webkit.org/show_bug.cgi?id=204417 InRadar P2 Normal --- 1 aakash_jain jbedard ap commit-queue jbedard simon.fraser webkit-bot-watchers-bugzilla webkit-bug-importer oldest_to_newest 1591971 0 aakash_jain 2019-11-20 05:14:37 -0800 If the layout-test finds any test failure, we re-run the tests (to rule-out flakiness) and do clean-tree run (to check pre-existing failures). on iOS, recently the re-runs and clean-tree runs have frequently started failing with 30+ failures. e.g.: https://ews-build.webkit.org/#/builders/24/builds/5262 https://ews-build.webkit.org/#/builders/24/builds/5260 https://ews-build.webkit.org/#/builders/24/builds/5259 https://ews-build.webkit.org/#/builders/24/builds/5256 https://ews-build.webkit.org/#/builders/24/builds/5251 https://ews-build.webkit.org/#/builders/24/builds/5077 https://ews-build.webkit.org/#/builders/24/builds/4981 https://ews-build.webkit.org/#/builders/24/builds/4779 This behavior is somewhat flaky. For e.g. in https://ews-build.webkit.org/#/builders/24/builds/5261 run-layout-tests-without-patch worked, and in https://ews-build.webkit.org/#/builders/24/builds/5254 re-run-layout-tests worked. 1591972 1 aakash_jain 2019-11-20 05:15:04 -0800 This issue stalls iOS WK2 queue whenever there is any failure on trunk. 1592012 2 jbedard 2019-11-20 08:59:53 -0800 I don't understand how clean-tree results are somehow more flakey than the test runs with the patch. 1592017 3 aakash_jain 2019-11-20 09:06:50 -0800 This seems like another fallout from Catalina update on iOS bots. We updated the bots around Oct 30, and we started seeing 30+ layout-test failure in https://bugs.webkit.org/show_bug.cgi?id=203792 That issue was fixed on Nov 10. I believe this issue has been happening since then. 1592045 4 jbedard 2019-11-20 10:27:14 -0800 I bet that running kill-old-processes between runs would help. As Alexey pointed out, this is probably a bug of some sort, so I just landed <https://trac.webkit.org/r252691> to help us figure out what the problem is. 1592073 5 383974 jbedard 2019-11-20 11:13:58 -0800 Created attachment 383974 Patch 1592086 6 383974 jbedard 2019-11-20 11:56:00 -0800 Comment on attachment 383974 Patch Now that we have some logging from <https://trac.webkit.org/r252691>, it seems to be simulator and https. I'll work on fixes for those problems independently, but as I mention offline to Aakash: "I think the kill-old-processes stuff needs to land. If nothing else, it's possible that a patch which fails to clean up resources appropriately could cause failures in the clean run and cause us to mark the patch as 'good' even though it is demonstrably not" 1592098 7 jbedard 2019-11-20 12:14:28 -0800 Committed r252700: <https://trac.webkit.org/changeset/252700> 1592099 8 383974 jbedard 2019-11-20 12:15:09 -0800 Comment on attachment 383974 Patch Landed manually, removing flags. 1592100 9 webkit-bug-importer 2019-11-20 12:15:19 -0800 <rdar://problem/57368402> 1592101 10 webkit-bug-importer 2019-11-20 12:15:25 -0800 <rdar://problem/57368404> 383974 2019-11-20 11:13:58 -0800 2019-11-20 12:15:09 -0800 Patch bug-204409-20191120111358.patch text/plain 4982 jbedard SW5kZXg6IFRvb2xzL0NoYW5nZUxvZwo9PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09Ci0tLSBUb29scy9DaGFuZ2VMb2cJKHJl dmlzaW9uIDI1MjY5NikKKysrIFRvb2xzL0NoYW5nZUxvZwkod29ya2luZyBjb3B5KQpAQCAtMSwz ICsxLDE5IEBACisyMDE5LTExLTIwICBKb25hdGhhbiBCZWRhcmQgIDxqYmVkYXJkQGFwcGxlLmNv bT4KKworICAgICAgICBbRVdTXSBpT1MgbGF5b3V0LXRlc3RzIGZhaWxzIHdpdGggMzArIGZhaWx1 cmVzIG9uIHJlLXJ1biBhbmQgb24gY2xlYW4tdHJlZQorICAgICAgICBodHRwczovL2J1Z3Mud2Vi a2l0Lm9yZy9zaG93X2J1Zy5jZ2k/aWQ9MjA0NDA5CisKKyAgICAgICAgUmV2aWV3ZWQgYnkgTk9C T0RZIChPT1BTISkuCisKKyAgICAgICAgKiBCdWlsZFNsYXZlU3VwcG9ydC9ld3MtYnVpbGQvc3Rl cHMucHk6CisgICAgICAgIChSdW5XZWJLaXRQZXJsVGVzdHMuZXZhbHVhdGVDb21tYW5kKTogQWRk IGtpbGxPbGRQcm9jZXNzZXMgYmVmb3JlIHJ1bm5pbmcgdGVzdCBzdWl0ZS4KKyAgICAgICAgKFJ1 bkphdmFTY3JpcHRDb3JlVGVzdHMuZXZhbHVhdGVDb21tYW5kKTogRGl0dG8uCisgICAgICAgIChS ZVJ1bkphdmFTY3JpcHRDb3JlVGVzdHMuZXZhbHVhdGVDb21tYW5kKTogRGl0dG8uCisgICAgICAg IChSdW5XZWJLaXRUZXN0cy5ldmFsdWF0ZUNvbW1hbmQpOiBEaXR0by4KKyAgICAgICAgKFJlUnVu V2ViS2l0VGVzdHMuZXZhbHVhdGVDb21tYW5kKTogRGl0dG8uCisgICAgICAgIChSdW5BUElUZXN0 cy5ldmFsdWF0ZUNvbW1hbmQpOiBEaXR0by4KKyAgICAgICAgKFJlUnVuQVBJVGVzdHMuZXZhbHVh dGVDb21tYW5kKTogRGl0dG8uCisKIDIwMTktMTEtMjAgIEpvbmF0aGFuIEJlZGFyZCAgPGpiZWRh cmRAYXBwbGUuY29tPgogCiAgICAgICAgIGtpbGwtb2xkLXByb2Nlc3NlczogTG9nIGN1cnJlbnRs eSBydW5uaW5nIHByb2Nlc3NlcwpJbmRleDogVG9vbHMvQnVpbGRTbGF2ZVN1cHBvcnQvZXdzLWJ1 aWxkL3N0ZXBzLnB5Cj09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT0KLS0tIFRvb2xzL0J1aWxkU2xhdmVTdXBwb3J0L2V3cy1i dWlsZC9zdGVwcy5weQkocmV2aXNpb24gMjUyNjkzKQorKysgVG9vbHMvQnVpbGRTbGF2ZVN1cHBv cnQvZXdzLWJ1aWxkL3N0ZXBzLnB5CSh3b3JraW5nIGNvcHkpCkBAIC02MzIsNyArNjMyLDcgQEAg Y2xhc3MgUnVuV2ViS2l0UGVybFRlc3RzKHNoZWxsLlNoZWxsQ29tbQogICAgIGRlZiBldmFsdWF0 ZUNvbW1hbmQoc2VsZiwgY21kKToKICAgICAgICAgcmMgPSBzaGVsbC5TaGVsbENvbW1hbmQuZXZh bHVhdGVDb21tYW5kKHNlbGYsIGNtZCkKICAgICAgICAgaWYgcmMgPT0gRkFJTFVSRToKLSAgICAg ICAgICAgIHNlbGYuYnVpbGQuYWRkU3RlcHNBZnRlckN1cnJlbnRTdGVwKFtSZVJ1bldlYktpdFBl cmxUZXN0cygpXSkKKyAgICAgICAgICAgIHNlbGYuYnVpbGQuYWRkU3RlcHNBZnRlckN1cnJlbnRT dGVwKFtLaWxsT2xkUHJvY2Vzc2VzKCksIFJlUnVuV2ViS2l0UGVybFRlc3RzKCldKQogICAgICAg ICByZXR1cm4gcmMKIAogCkBAIC05NzQsNyArOTc0LDcgQEAgY2xhc3MgUnVuSmF2YVNjcmlwdENv cmVUZXN0cyhzaGVsbC5UZXN0KQogICAgICAgICAgICAgc2VsZi5idWlsZC5yZXN1bHRzID0gU1VD Q0VTUwogICAgICAgICAgICAgc2VsZi5idWlsZC5idWlsZEZpbmlzaGVkKFttZXNzYWdlXSwgU1VD Q0VTUykKICAgICAgICAgZWxzZToKLSAgICAgICAgICAgIHNlbGYuYnVpbGQuYWRkU3RlcHNBZnRl ckN1cnJlbnRTdGVwKFtWYWxpZGF0ZVBhdGNoKHZlcmlmeUJ1Z0Nsb3NlZD1GYWxzZSwgYWRkVVJM cz1GYWxzZSksIFJlUnVuSmF2YVNjcmlwdENvcmVUZXN0cygpXSkKKyAgICAgICAgICAgIHNlbGYu YnVpbGQuYWRkU3RlcHNBZnRlckN1cnJlbnRTdGVwKFtWYWxpZGF0ZVBhdGNoKHZlcmlmeUJ1Z0Ns b3NlZD1GYWxzZSwgYWRkVVJMcz1GYWxzZSksIEtpbGxPbGRQcm9jZXNzZXMoKSwgUmVSdW5KYXZh U2NyaXB0Q29yZVRlc3RzKCldKQogICAgICAgICByZXR1cm4gcmMKIAogICAgIGRlZiBjb21tYW5k Q29tcGxldGUoc2VsZiwgY21kKToKQEAgLTEwNTAsNiArMTA1MCw3IEBAIGNsYXNzIFJlUnVuSmF2 YVNjcmlwdENvcmVUZXN0cyhSdW5KYXZhU2MKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgIFZhbGlkYXRlUGF0Y2godmVyaWZ5QnVnQ2xvc2VkPUZhbHNlLCBh ZGRVUkxzPUZhbHNlKSwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgIENvbXBpbGVKU0NUb1QoKSwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgIFZhbGlkYXRlUGF0Y2godmVyaWZ5QnVnQ2xvc2VkPUZhbHNlLCBhZGRV UkxzPUZhbHNlKSwKKyAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgIEtpbGxPbGRQcm9jZXNzZXMoKSwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgIFJ1bkpTQ1Rlc3RzV2l0aG91dFBhdGNoKCksCiAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBBbmFseXplSlNDVGVzdHNSZXN1bHRz KCldKQogICAgICAgICByZXR1cm4gcmMKQEAgLTEyODgsNyArMTI4OSwxNCBAQCBjbGFzcyBSdW5X ZWJLaXRUZXN0cyhzaGVsbC5UZXN0KToKICAgICAgICAgICAgIHNlbGYuYnVpbGQucmVzdWx0cyA9 IFNVQ0NFU1MKICAgICAgICAgICAgIHNlbGYuYnVpbGQuYnVpbGRGaW5pc2hlZChbbWVzc2FnZV0s IFNVQ0NFU1MpCiAgICAgICAgIGVsc2U6Ci0gICAgICAgICAgICBzZWxmLmJ1aWxkLmFkZFN0ZXBz QWZ0ZXJDdXJyZW50U3RlcChbQXJjaGl2ZVRlc3RSZXN1bHRzKCksIFVwbG9hZFRlc3RSZXN1bHRz KCksIEV4dHJhY3RUZXN0UmVzdWx0cygpLCBWYWxpZGF0ZVBhdGNoKHZlcmlmeUJ1Z0Nsb3NlZD1G YWxzZSwgYWRkVVJMcz1GYWxzZSksIFJlUnVuV2ViS2l0VGVzdHMoKV0pCisgICAgICAgICAgICBz ZWxmLmJ1aWxkLmFkZFN0ZXBzQWZ0ZXJDdXJyZW50U3RlcChbCisgICAgICAgICAgICAgICAgQXJj aGl2ZVRlc3RSZXN1bHRzKCksCisgICAgICAgICAgICAgICAgVXBsb2FkVGVzdFJlc3VsdHMoKSwK KyAgICAgICAgICAgICAgICBFeHRyYWN0VGVzdFJlc3VsdHMoKSwKKyAgICAgICAgICAgICAgICBW YWxpZGF0ZVBhdGNoKHZlcmlmeUJ1Z0Nsb3NlZD1GYWxzZSwgYWRkVVJMcz1GYWxzZSksCisgICAg ICAgICAgICAgICAgS2lsbE9sZFByb2Nlc3NlcygpLAorICAgICAgICAgICAgICAgIFJlUnVuV2Vi S2l0VGVzdHMoKSwKKyAgICAgICAgICAgIF0pCiAgICAgICAgIHJldHVybiByYwogCiAgICAgZGVm IGdldFJlc3VsdFN1bW1hcnkoc2VsZik6CkBAIC0xMzIwLDYgKzEzMjgsNyBAQCBjbGFzcyBSZVJ1 bldlYktpdFRlc3RzKFJ1bldlYktpdFRlc3RzKToKICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgIFZhbGlkYXRlUGF0Y2godmVyaWZ5QnVnQ2xvc2VkPUZhbHNl LCBhZGRVUkxzPUZhbHNlKSwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgIENvbXBpbGVXZWJLaXRUb1QoKSwKICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgIFZhbGlkYXRlUGF0Y2godmVyaWZ5QnVnQ2xvc2VkPUZhbHNl LCBhZGRVUkxzPUZhbHNlKSwKKyAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgIEtpbGxPbGRQcm9jZXNzZXMoKSwKICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgIFJ1bldlYktpdFRlc3RzV2l0aG91dFBhdGNoKCldKQogICAg ICAgICByZXR1cm4gcmMKIApAQCAtMTYyNiw3ICsxNjM1LDcgQEAgY2xhc3MgUnVuQVBJVGVzdHMo VGVzdFdpdGhGYWlsdXJlQ291bnQpOgogICAgICAgICAgICAgc2VsZi5idWlsZC5yZXN1bHRzID0g U1VDQ0VTUwogICAgICAgICAgICAgc2VsZi5idWlsZC5idWlsZEZpbmlzaGVkKFttZXNzYWdlXSwg U1VDQ0VTUykKICAgICAgICAgZWxzZToKLSAgICAgICAgICAgIHNlbGYuYnVpbGQuYWRkU3RlcHNB ZnRlckN1cnJlbnRTdGVwKFtWYWxpZGF0ZVBhdGNoKHZlcmlmeUJ1Z0Nsb3NlZD1GYWxzZSwgYWRk VVJMcz1GYWxzZSksIFJlUnVuQVBJVGVzdHMoKV0pCisgICAgICAgICAgICBzZWxmLmJ1aWxkLmFk ZFN0ZXBzQWZ0ZXJDdXJyZW50U3RlcChbVmFsaWRhdGVQYXRjaCh2ZXJpZnlCdWdDbG9zZWQ9RmFs c2UsIGFkZFVSTHM9RmFsc2UpLCBLaWxsT2xkUHJvY2Vzc2VzKCksIFJlUnVuQVBJVGVzdHMoKV0p CiAgICAgICAgIHJldHVybiByYwogCiAKQEAgLTE2NDYsNiArMTY1NSw3IEBAIGNsYXNzIFJlUnVu QVBJVGVzdHMoUnVuQVBJVGVzdHMpOgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgVmFsaWRhdGVQYXRjaCh2ZXJpZnlCdWdDbG9zZWQ9RmFsc2UsIGFkZFVS THM9RmFsc2UpLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgQ29tcGlsZVdlYktpdFRvVCgpLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgVmFsaWRhdGVQYXRjaCh2ZXJpZnlCdWdDbG9zZWQ9RmFsc2UsIGFkZFVS THM9RmFsc2UpLAorICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgS2lsbE9sZFByb2Nlc3NlcygpLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgUnVuQVBJVGVzdHNXaXRob3V0UGF0Y2goKSwKICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIEFuYWx5emVBUElUZXN0c1Jlc3VsdHMo KV0pCiAgICAgICAgIHJldHVybiByYwo=