172322 2017-05-18 15:52:17 -0700 webkitpy: Layout tests which have no output will succeed 2017-05-30 07:55:16 -0700 1 1 1 Unclassified WebKit Tools / Tests WebKit Nightly Build Unspecified Unspecified RESOLVED FIXED https://bugs.webkit.org/show_bug.cgi?id=172378 P2 Normal --- 1 jbedard jbedard buildbot commit-queue dbates ddkilzer glenn lforschler oldest_to_newest 1310312 0 jbedard 2017-05-18 15:52:17 -0700 If a layout test has no output, it will succeed. Such tests should fail. 1310313 1 jbedard 2017-05-18 15:53:00 -0700 Tracked internally at <rdar://problem/32258350> 1310317 2 310568 jbedard 2017-05-18 15:56:11 -0700 Created attachment 310568 Patch 1310321 3 310568 ddkilzer 2017-05-18 16:00:22 -0700 Comment on attachment 310568 Patch r=me 1310322 4 310568 jbedard 2017-05-18 16:03:21 -0700 Comment on attachment 310568 Patch Waiting till EWS comes back before landing. 1310620 5 310568 commit-queue 2017-05-19 09:23:18 -0700 Comment on attachment 310568 Patch Clearing flags on attachment: 310568 Committed r217130: <http://trac.webkit.org/changeset/217130> 1310621 6 commit-queue 2017-05-19 09:23:19 -0700 All reviewed patches have been landed. Closing bug. 1310976 7 dbates 2017-05-19 22:53:14 -0700 Should we take a similar approach in SingleTestRunner._compare_audio() when there is no actual result? 1310977 8 dbates 2017-05-19 22:57:36 -0700 How are these tests classified in the layout test results results.html file? 1310978 9 dbates 2017-05-19 23:05:31 -0700 (In reply to Daniel Bates from comment #8) > How are these tests classified in the layout test results results.html file? They are classified as test_expectations.MISSING in determine_result_type() (defined in Tools/Scripts/webkitpy/layout_tests/models/test_failures.py). Tests that have no actual result more than likely represent a bug in the test tool/infrastructure and we should segregate them from tests that are missing their expected results. 1310979 10 310568 dbates 2017-05-19 23:07:14 -0700 Comment on attachment 310568 Patch View in context: https://bugs.webkit.org/attachment.cgi?id=310568&action=review > Tools/Scripts/webkitpy/layout_tests/models/test_failures.py:170 > + return 'test was not run' Is this correct? I mean, did we really not run the test or did we run the test and it had no output? 1313719 11 jbedard 2017-05-30 07:55:16 -0700 (In reply to Daniel Bates from comment #10) > Comment on attachment 310568 [details] > Patch > > View in context: > https://bugs.webkit.org/attachment.cgi?id=310568&action=review > > > Tools/Scripts/webkitpy/layout_tests/models/test_failures.py:170 > > + return 'test was not run' > > Is this correct? I mean, did we really not run the test or did we run the > test and it had no output? We attempted to run the test and had no output. This could mean that we actually ran the test and had to output, or it could mean that the test was not run at all. In my experience, this failure is usually caused by a test failing to run, although, the tests associated with https://bugs.webkit.org/show_bug.cgi?id=172378 likely have different causes. 310568 2017-05-18 15:56:11 -0700 2017-05-19 09:23:18 -0700 Patch bug-172322-20170518155610.patch text/plain 3413 jbedard SW5kZXg6IFRvb2xzL0NoYW5nZUxvZwo9PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09Ci0tLSBUb29scy9DaGFuZ2VMb2cJKHJl dmlzaW9uIDIxNzA3MikKKysrIFRvb2xzL0NoYW5nZUxvZwkod29ya2luZyBjb3B5KQpAQCAtMSwz ICsxLDE5IEBACisyMDE3LTA1LTE4ICBKb25hdGhhbiBCZWRhcmQgIDxqYmVkYXJkQGFwcGxlLmNv bT4KKworICAgICAgICB3ZWJraXRweTogTGF5b3V0IHRlc3RzIHdoaWNoIGhhdmUgbm8gb3V0cHV0 IHdpbGwgc3VjY2VlZAorICAgICAgICBodHRwczovL2J1Z3Mud2Via2l0Lm9yZy9zaG93X2J1Zy5j Z2k/aWQ9MTcyMzIyCisgICAgICAgIDxyZGFyOi8vcHJvYmxlbS8zMjI1ODM1MD4KKworICAgICAg ICBSZXZpZXdlZCBieSBOT0JPRFkgKE9PUFMhKS4KKworICAgICAgICAqIFNjcmlwdHMvd2Via2l0 cHkvbGF5b3V0X3Rlc3RzL2NvbnRyb2xsZXJzL3NpbmdsZV90ZXN0X3J1bm5lci5weToKKyAgICAg ICAgKFNpbmdsZVRlc3RSdW5uZXIuX2NvbXBhcmVfdGV4dCk6IEFkZCBGYWlscmVOb3RUZXN0ZWQg dG8gZmFpbHVyZXMgaWYgdGVzdCBoYXMgbm8gb3V0cHV0LgorICAgICAgICAqIFNjcmlwdHMvd2Vi a2l0cHkvbGF5b3V0X3Rlc3RzL21vZGVscy90ZXN0X2ZhaWx1cmVzLnB5OgorICAgICAgICAoZGV0 ZXJtaW5lX3Jlc3VsdF90eXBlKTogRmFpbHVyZU5vdFRlc3RlZCBtZWFucyB0aGF0IHRoZSB0ZXN0 IGNvdWxkIG5vdCBiZSBsb2FkZWQuIFRoaXMKKyAgICAgICAgaXMgY2xhc3NpZmllZCBhcyBhIE1J U1NJTkcgZmFpbHVyZS4KKyAgICAgICAgKEZhaWx1cmVOb3RUZXN0ZWQpOiBBZGRlZC4KKyAgICAg ICAgKEZhaWx1cmVOb3RUZXN0ZWQubWVzc2FnZSk6CisKIDIwMTctMDUtMTggIEpvaG4gV2lsYW5k ZXIgIDx3aWxhbmRlckBhcHBsZS5jb20+CiAKICAgICAgICAgUmVzb3VyY2UgTG9hZCBTdGF0aXN0 aWNzOiBHcmFuZGZhdGhlciBkb21haW5zIGZvciBleGlzdGluZyBkYXRhIHJlY29yZHMKSW5kZXg6 IFRvb2xzL1NjcmlwdHMvd2Via2l0cHkvbGF5b3V0X3Rlc3RzL2NvbnRyb2xsZXJzL3NpbmdsZV90 ZXN0X3J1bm5lci5weQo9PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09Ci0tLSBUb29scy9TY3JpcHRzL3dlYmtpdHB5L2xheW91 dF90ZXN0cy9jb250cm9sbGVycy9zaW5nbGVfdGVzdF9ydW5uZXIucHkJKHJldmlzaW9uIDIxNzA0 NikKKysrIFRvb2xzL1NjcmlwdHMvd2Via2l0cHkvbGF5b3V0X3Rlc3RzL2NvbnRyb2xsZXJzL3Np bmdsZV90ZXN0X3J1bm5lci5weQkod29ya2luZyBjb3B5KQpAQCAtMjQwLDYgKzI0MCw4IEBAIGNs YXNzIFNpbmdsZVRlc3RSdW5uZXIob2JqZWN0KToKICAgICAgICAgICAgIGZhaWx1cmVzLmFwcGVu ZCh0ZXN0X2ZhaWx1cmVzLkZhaWx1cmVUZXh0TWlzbWF0Y2goKSkKICAgICAgICAgZWxpZiBhY3R1 YWxfdGV4dCBhbmQgbm90IGV4cGVjdGVkX3RleHQ6CiAgICAgICAgICAgICBmYWlsdXJlcy5hcHBl bmQodGVzdF9mYWlsdXJlcy5GYWlsdXJlTWlzc2luZ1Jlc3VsdCgpKQorICAgICAgICBlbGlmIG5v dCBhY3R1YWxfdGV4dCBhbmQgZXhwZWN0ZWRfdGV4dDoKKyAgICAgICAgICAgIGZhaWx1cmVzLmFw cGVuZCh0ZXN0X2ZhaWx1cmVzLkZhaWx1cmVOb3RUZXN0ZWQoKSkKICAgICAgICAgcmV0dXJuIGZh aWx1cmVzCiAKICAgICBkZWYgX2NvbXBhcmVfYXVkaW8oc2VsZiwgZXhwZWN0ZWRfYXVkaW8sIGFj dHVhbF9hdWRpbyk6CkluZGV4OiBUb29scy9TY3JpcHRzL3dlYmtpdHB5L2xheW91dF90ZXN0cy9t b2RlbHMvdGVzdF9mYWlsdXJlcy5weQo9PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09Ci0tLSBUb29scy9TY3JpcHRzL3dlYmtp dHB5L2xheW91dF90ZXN0cy9tb2RlbHMvdGVzdF9mYWlsdXJlcy5weQkocmV2aXNpb24gMjE3MDQ2 KQorKysgVG9vbHMvU2NyaXB0cy93ZWJraXRweS9sYXlvdXRfdGVzdHMvbW9kZWxzL3Rlc3RfZmFp bHVyZXMucHkJKHdvcmtpbmcgY29weSkKQEAgLTYyLDcgKzYyLDggQEAgZGVmIGRldGVybWluZV9y ZXN1bHRfdHlwZShmYWlsdXJlX2xpc3QpOgogICAgIGVsaWYgKEZhaWx1cmVNaXNzaW5nUmVzdWx0 IGluIGZhaWx1cmVfdHlwZXMgb3IKICAgICAgICAgICBGYWlsdXJlTWlzc2luZ0ltYWdlIGluIGZh aWx1cmVfdHlwZXMgb3IKICAgICAgICAgICBGYWlsdXJlTWlzc2luZ0ltYWdlSGFzaCBpbiBmYWls dXJlX3R5cGVzIG9yCi0gICAgICAgICAgRmFpbHVyZU1pc3NpbmdBdWRpbyBpbiBmYWlsdXJlX3R5 cGVzKToKKyAgICAgICAgICBGYWlsdXJlTWlzc2luZ0F1ZGlvIGluIGZhaWx1cmVfdHlwZXMgb3IK KyAgICAgICAgICBGYWlsdXJlTm90VGVzdGVkIGluIGZhaWx1cmVfdHlwZXMpOgogICAgICAgICBy ZXR1cm4gdGVzdF9leHBlY3RhdGlvbnMuTUlTU0lORwogICAgIGVsc2U6CiAgICAgICAgIGlzX3Rl eHRfZmFpbHVyZSA9IEZhaWx1cmVUZXh0TWlzbWF0Y2ggaW4gZmFpbHVyZV90eXBlcwpAQCAtMTY0 LDYgKzE2NSwxMCBAQCBjbGFzcyBGYWlsdXJlTWlzc2luZ1Jlc3VsdChGYWlsdXJlVGV4dCk6CiAg ICAgICAgIHJldHVybiAiLWV4cGVjdGVkLnR4dCB3YXMgbWlzc2luZyIKIAogCitjbGFzcyBGYWls dXJlTm90VGVzdGVkKEZhaWx1cmVUZXh0KToKKyAgICBkZWYgbWVzc2FnZShzZWxmKToKKyAgICAg ICAgcmV0dXJuICd0ZXN0IHdhcyBub3QgcnVuJworCiBjbGFzcyBGYWlsdXJlVGV4dE1pc21hdGNo KEZhaWx1cmVUZXh0KToKICAgICBkZWYgbWVzc2FnZShzZWxmKToKICAgICAgICAgcmV0dXJuICJ0 ZXh0IGRpZmYiCkBAIC0yNjUsNyArMjcwLDcgQEAgY2xhc3MgRmFpbHVyZUVhcmx5RXhpdChUZXN0 RmFpbHVyZSk6CiAKICMgQ29udmVuaWVudCBjb2xsZWN0aW9uIG9mIGFsbCBmYWlsdXJlIGNsYXNz ZXMgZm9yIGFueXRoaW5nIHRoYXQgbWlnaHQKICMgbmVlZCB0byBlbnVtZXJhdGUgb3ZlciB0aGVt IGFsbC4KLUFMTF9GQUlMVVJFX0NMQVNTRVMgPSAoRmFpbHVyZVRpbWVvdXQsIEZhaWx1cmVDcmFz aCwgRmFpbHVyZU1pc3NpbmdSZXN1bHQsCitBTExfRkFJTFVSRV9DTEFTU0VTID0gKEZhaWx1cmVU aW1lb3V0LCBGYWlsdXJlQ3Jhc2gsIEZhaWx1cmVNaXNzaW5nUmVzdWx0LCBGYWlsdXJlTm90VGVz dGVkLAogICAgICAgICAgICAgICAgICAgICAgICBGYWlsdXJlVGV4dE1pc21hdGNoLCBGYWlsdXJl TWlzc2luZ0ltYWdlSGFzaCwKICAgICAgICAgICAgICAgICAgICAgICAgRmFpbHVyZU1pc3NpbmdJ bWFnZSwgRmFpbHVyZUltYWdlSGFzaE1pc21hdGNoLAogICAgICAgICAgICAgICAgICAgICAgICBG YWlsdXJlSW1hZ2VIYXNoSW5jb3JyZWN0LCBGYWlsdXJlUmVmdGVzdE1pc21hdGNoLAo=