58858 2011-04-18 18:48:26 -0700 reftest mismatch doesn't work 2011-08-15 04:19:12 -0700 1 1 1 Unclassified WebKit Tools / Tests 528+ (Nightly build) PC OS X 10.5 RESOLVED WONTFIX P2 Normal --- 59188 1 ojan webkit-unassigned dimich dpranke hayato ojan oldest_to_newest 388205 0 ojan 2011-04-18 18:48:26 -0700 I'm probably just doing it wrong, but I couldn't fine any sample expected mismatch reftests to compare to. I made foo.html and foo-expected-mismatch.html and made their contents identical (just "asdf"). The test passed, but should have failed, right? 388212 1 dpranke 2011-04-18 18:59:00 -0700 reftests only apply to the image part of the test, not the text part. 388214 2 dpranke 2011-04-18 18:59:55 -0700 there are (artificial) samples in webkitpy.layout_tests.port/test.py that you can play with via new-run-webkit-tests --platform test, if that's helpful. Of course, they don't generate real output, so it probably isn't for what you're trying to do. 388218 3 ojan 2011-04-18 19:04:57 -0700 (In reply to comment #1) > reftests only apply to the image part of the test, not the text part. But still, the test should pass, right? I was running with -p. (In reply to comment #2) > there are (artificial) samples in webkitpy.layout_tests.port/test.py that you can play with via new-run-webkit-tests --platform test, if that's helpful. Of course, they don't generate real output, so it probably isn't for what you're trying to do. I'll take a look. 388244 4 dpranke 2011-04-18 19:35:00 -0700 (In reply to comment #3) > (In reply to comment #1) > > reftests only apply to the image part of the test, not the text part. > > But still, the test should pass, right? I was running with -p. > Oh, I see, I read that as you had made the text output the same. Sorry. Yes, I would expect such a test to fail. 388249 5 hayato 2011-04-18 19:56:31 -0700 I'll take a look. There is no sample 'real' mismatch reftests in LayoutTest directory. So I might miss something. I'll try to submit a sample mismatch reftest, expecting to find a issue. (In reply to comment #4) > (In reply to comment #3) > > (In reply to comment #1) > > > reftests only apply to the image part of the test, not the text part. > > > > But still, the test should pass, right? I was running with -p. > > > > Oh, I see, I read that as you had made the text output the same. Sorry. Yes, I would expect such a test to fail. 388290 6 90142 hayato 2011-04-18 21:42:22 -0700 Created attachment 90142 add-sample-reftests 388291 7 hayato 2011-04-18 21:45:42 -0700 I posted a patch which adds two sample reftests to the fast/harness directory. It seems a mismatch-reftest works correctly on my environment. I cannot reproduce the wrong behavior. Hmm. It would be great if you could help finding an issue about mismatch reftests. (In reply to comment #6) > Created an attachment (id=90142) [details] > add-sample-reftests 389400 8 90142 dimich 2011-04-20 11:54:34 -0700 Comment on attachment 90142 add-sample-reftests When I aply this patch and run: build-webkit new-run-webkit-tests LayoutTests/fast/harness/ I get: Regressions: Unexpected image mismatch : (2) fast/harness/sample-fail-mismatch-reftest.html = IMAGE fast/harness/sample-mismatch-reftest.html = IMAGE The images are similar for first test and different for the second, but they both fail due to image mismatch. Note this is not a Chromium DRT, it's the regular mac build of WebKit. I would expect only one of those to fail, and also it would be nice to add the 'expected to fail' test to other platform's test_expectations.txt file. 389844 9 hayato 2011-04-20 22:06:04 -0700 Hi Dmitry, Thank you for testing that. I can reproduce that on mac build of WebKit. I'll take a look further. Sorry for the inconvenience. (In reply to comment #8) > (From update of attachment 90142 [details]) > When I aply this patch and run: > > build-webkit > new-run-webkit-tests LayoutTests/fast/harness/ > > I get: > > Regressions: Unexpected image mismatch : (2) > fast/harness/sample-fail-mismatch-reftest.html = IMAGE > fast/harness/sample-mismatch-reftest.html = IMAGE > > The images are similar for first test and different for the second, but they both fail due to image mismatch. > Note this is not a Chromium DRT, it's the regular mac build of WebKit. > > I would expect only one of those to fail, and also it would be nice to add the 'expected to fail' test to other platform's test_expectations.txt file. (In reply to comment #8) > (From update of attachment 90142 [details]) > When I aply this patch and run: > > build-webkit > new-run-webkit-tests LayoutTests/fast/harness/ > > I get: > > Regressions: Unexpected image mismatch : (2) > fast/harness/sample-fail-mismatch-reftest.html = IMAGE > fast/harness/sample-mismatch-reftest.html = IMAGE > > The images are similar for first test and different for the second, but they both fail due to image mismatch. > Note this is not a Chromium DRT, it's the regular mac build of WebKit. > > I would expect only one of those to fail, and also it would be nice to add the 'expected to fail' test to other platform's test_expectations.txt file. 389868 10 hayato 2011-04-20 23:29:49 -0700 Let me note what I figured out: - WebKitPort, which is used in new-run-webkit-tests, is trying to disable 'pixel_tests' in default. - We should skip reftests if pixel_tests is disabled. - Even if '--pixel-tests' in turned on explicitly on WebKitPort, there is a difference between ChromiumLinuxDriver and WebKitDriver about how to handle 'image_hash' value. Such a different behavior makes a correct reftests fail on Mac build of WebKit. I'll file bugs for each. 390708 11 hayato 2011-04-22 04:35:37 -0700 I filed the two bugs: - https://bugs.webkit.org/show_bug.cgi?id=59078 - https://bugs.webkit.org/show_bug.cgi?id=59188 417937 12 hayato 2011-06-09 03:37:40 -0700 I posted a patch to fix a WebKitDriver so that reftests work in bug 59188. After the patch in bug 59188 is landed, I am going to add these sample reftests too. 420257 13 hayato 2011-06-14 00:24:22 -0700 Bug 59188 was landed. I'd like to commit these sample mismatch reftests to detect a regression. Should I rename the subject of this bugzilla entry or create a new bug for these sample tests? (In reply to comment #12) > I posted a patch to fix a WebKitDriver so that reftests work in bug 59188. > After the patch in bug 59188 is landed, I am going to add these sample reftests too. 451029 14 hayato 2011-08-15 04:18:42 -0700 It might be better to close this bug. A mismatch refrests should work on all platfoms after fixing bug 59188. I'll file a separate bug for adding sample mismatch reftests. 90142 2011-04-18 21:42:22 -0700 2011-08-15 04:19:12 -0700 add-sample-reftests bug-58858-20110419134220.patch text/plain 3578 hayato U3VidmVyc2lvbiBSZXZpc2lvbjogODQxMzQKZGlmZiAtLWdpdCBhL0xheW91dFRlc3RzL0NoYW5n ZUxvZyBiL0xheW91dFRlc3RzL0NoYW5nZUxvZwppbmRleCAwYWZjNmZiOGY5ZmU4Yjk4MjFiMWIx N2ExMzAwZmJhNjc2Yzg4MzVlLi5mOWVkMGIwZTI4NzViN2IzMDZkYjM2YThhZDBkZDUwNGQ3NWE3 NzJlIDEwMDY0NAotLS0gYS9MYXlvdXRUZXN0cy9DaGFuZ2VMb2cKKysrIGIvTGF5b3V0VGVzdHMv Q2hhbmdlTG9nCkBAIC0xLDMgKzEsMjAgQEAKKzIwMTEtMDQtMTggIEhheWF0byBJdG8gIDxoYXlh dG9AY2hyb21pdW0ub3JnPgorCisgICAgICAgIFJldmlld2VkIGJ5IE5PQk9EWSAoT09QUyEpLgor CisgICAgICAgIEFkZCB0d28gc2FtcGxlIG1pc21hdGNoIHJlZnRlc3RzIHRvIG1ha2Ugc3VyZSB0 aGF0IHJlZnRlc3RzIHdvcmsgY29ycmVjdGx5LgorCisgICAgICAgIE9uZSBtaXNtYXRjaCByZWZ0 ZXN0IGlzIGV4cGVjdGVkIHRvIHBhc3MsIHRoZSBvdGhlciBpcyBleHBlY3RlZCB0byBmYWlsLgor ICAgICAgICBUaGUgZmFpbGVkIG9uZSBpcyBtYXJrZWQgYXMgJ1dPTlRGSVgnIGluIHRlc3QgZXhw ZWN0YXRpb25zLgorCisgICAgICAgIGh0dHBzOi8vYnVncy53ZWJraXQub3JnL3Nob3dfYnVnLmNn aT9pZD01ODg1OAorCisgICAgICAgICogZmFzdC9oYXJuZXNzL3NhbXBsZS1mYWlsLW1pc21hdGNo LXJlZnRlc3QtZXhwZWN0ZWQtbWlzbWF0Y2guaHRtbDogQWRkZWQuCisgICAgICAgICogZmFzdC9o YXJuZXNzL3NhbXBsZS1mYWlsLW1pc21hdGNoLXJlZnRlc3QuaHRtbDogQWRkZWQuCisgICAgICAg ICogZmFzdC9oYXJuZXNzL3NhbXBsZS1taXNtYXRjaC1yZWZ0ZXN0LWV4cGVjdGVkLW1pc21hdGNo Lmh0bWw6IEFkZGVkLgorICAgICAgICAqIGZhc3QvaGFybmVzcy9zYW1wbGUtbWlzbWF0Y2gtcmVm dGVzdC5odG1sOiBBZGRlZC4KKyAgICAgICAgKiBwbGF0Zm9ybS9jaHJvbWl1bS90ZXN0X2V4cGVj dGF0aW9ucy50eHQ6CisKIDIwMTEtMDQtMTggIEFuZHJleSBLb3N5YWtvdiAgPGNhc2VxQGNocm9t aXVtLm9yZz4KIAogICAgICAgICBVbnJldmlld2VkLiBBZGRlZCBjaHJvbWl1bSBiYXNlbGluZXMg Zm9yIG5ldyB0ZXN0cy4KZGlmZiAtLWdpdCBhL0xheW91dFRlc3RzL2Zhc3QvaGFybmVzcy9zYW1w bGUtZmFpbC1taXNtYXRjaC1yZWZ0ZXN0LWV4cGVjdGVkLW1pc21hdGNoLmh0bWwgYi9MYXlvdXRU ZXN0cy9mYXN0L2hhcm5lc3Mvc2FtcGxlLWZhaWwtbWlzbWF0Y2gtcmVmdGVzdC1leHBlY3RlZC1t aXNtYXRjaC5odG1sCm5ldyBmaWxlIG1vZGUgMTAwNjQ0CmluZGV4IDAwMDAwMDAwMDAwMDAwMDAw MDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAuLjdmODY0NGJkNGU2YTVkYTliYzZiYTBmM2M2N2E5ZDQ4 ZTBiMWRhMGUKLS0tIC9kZXYvbnVsbAorKysgYi9MYXlvdXRUZXN0cy9mYXN0L2hhcm5lc3Mvc2Ft cGxlLWZhaWwtbWlzbWF0Y2gtcmVmdGVzdC1leHBlY3RlZC1taXNtYXRjaC5odG1sCkBAIC0wLDAg KzEgQEAKK1RoaXMgbWlzbWF0Y2ggcmVmdGVzdCBpcyBleHBlY3RlZCB0byBmYWlsLgpkaWZmIC0t Z2l0IGEvTGF5b3V0VGVzdHMvZmFzdC9oYXJuZXNzL3NhbXBsZS1mYWlsLW1pc21hdGNoLXJlZnRl c3QuaHRtbCBiL0xheW91dFRlc3RzL2Zhc3QvaGFybmVzcy9zYW1wbGUtZmFpbC1taXNtYXRjaC1y ZWZ0ZXN0Lmh0bWwKbmV3IGZpbGUgbW9kZSAxMDA2NDQKaW5kZXggMDAwMDAwMDAwMDAwMDAwMDAw MDAwMDAwMDAwMDAwMDAwMDAwMDAwMC4uN2Y4NjQ0YmQ0ZTZhNWRhOWJjNmJhMGYzYzY3YTlkNDhl MGIxZGEwZQotLS0gL2Rldi9udWxsCisrKyBiL0xheW91dFRlc3RzL2Zhc3QvaGFybmVzcy9zYW1w bGUtZmFpbC1taXNtYXRjaC1yZWZ0ZXN0Lmh0bWwKQEAgLTAsMCArMSBAQAorVGhpcyBtaXNtYXRj aCByZWZ0ZXN0IGlzIGV4cGVjdGVkIHRvIGZhaWwuCmRpZmYgLS1naXQgYS9MYXlvdXRUZXN0cy9m YXN0L2hhcm5lc3Mvc2FtcGxlLW1pc21hdGNoLXJlZnRlc3QtZXhwZWN0ZWQtbWlzbWF0Y2guaHRt bCBiL0xheW91dFRlc3RzL2Zhc3QvaGFybmVzcy9zYW1wbGUtbWlzbWF0Y2gtcmVmdGVzdC1leHBl Y3RlZC1taXNtYXRjaC5odG1sCm5ldyBmaWxlIG1vZGUgMTAwNjQ0CmluZGV4IDAwMDAwMDAwMDAw MDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAuLmUwODBhNjQ5Y2I4YjY1ZDQ1Mjc0N2ZkODQ0 NGRmYTA5NmE3MDljMjEKLS0tIC9kZXYvbnVsbAorKysgYi9MYXlvdXRUZXN0cy9mYXN0L2hhcm5l c3Mvc2FtcGxlLW1pc21hdGNoLXJlZnRlc3QtZXhwZWN0ZWQtbWlzbWF0Y2guaHRtbApAQCAtMCww ICsxLDIgQEAKK1RoaXMgaXMgYSBzYW1wbGUgbWlzbWF0Y2ggcmVmdGVzdC4KK1RoaXMgbGluZSBl eGlzdHMgb25seSBpbiAnLWV4cGVjdGVkLW1pc21hdGNoLmh0bWwnLgpkaWZmIC0tZ2l0IGEvTGF5 b3V0VGVzdHMvZmFzdC9oYXJuZXNzL3NhbXBsZS1taXNtYXRjaC1yZWZ0ZXN0Lmh0bWwgYi9MYXlv dXRUZXN0cy9mYXN0L2hhcm5lc3Mvc2FtcGxlLW1pc21hdGNoLXJlZnRlc3QuaHRtbApuZXcgZmls ZSBtb2RlIDEwMDY0NAppbmRleCAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAw MDAwLi42MmQwZjQ4ZmFkYTQxN2M2NWQzZTRiY2JlNDIzODk2YjViODA0MmZmCi0tLSAvZGV2L251 bGwKKysrIGIvTGF5b3V0VGVzdHMvZmFzdC9oYXJuZXNzL3NhbXBsZS1taXNtYXRjaC1yZWZ0ZXN0 Lmh0bWwKQEAgLTAsMCArMSBAQAorVGhpcyBpcyBhIHNhbXBsZSBtaXNtYXRjaCByZWZ0ZXN0Lgpk aWZmIC0tZ2l0IGEvTGF5b3V0VGVzdHMvcGxhdGZvcm0vY2hyb21pdW0vdGVzdF9leHBlY3RhdGlv bnMudHh0IGIvTGF5b3V0VGVzdHMvcGxhdGZvcm0vY2hyb21pdW0vdGVzdF9leHBlY3RhdGlvbnMu dHh0CmluZGV4IDU1MWExMjVlZjUxNWFlMDQyMTQzZTBjYWI3YmU1MTU2NzdjZTljZTkuLmZmNTZl YzM1YmU1NWUxMzBmYTkwODI1YzA5N2Y0NDY4ODRkNmJkMzQgMTAwNjQ0Ci0tLSBhL0xheW91dFRl c3RzL3BsYXRmb3JtL2Nocm9taXVtL3Rlc3RfZXhwZWN0YXRpb25zLnR4dAorKysgYi9MYXlvdXRU ZXN0cy9wbGF0Zm9ybS9jaHJvbWl1bS90ZXN0X2V4cGVjdGF0aW9ucy50eHQKQEAgLTQ5NSw2ICs0 OTUsOSBAQCBXT05URklYIFNLSVAgOiBmYXN0L3BhcnNlci9wcmUtaHRtbDUtcGFyc2VyLXF1aXJr cy5odG1sID0gRkFJTAogLy8gVGhlIHRlc3QgZGVwZW5kcyBvbiBtZXRyaWNzIG9uIG5vbi1NYWMu CiBXT05URklYIE1BQyA6IGZhc3QvZm9ybXMvaW5wdXQtbnVtYmVyLWxhcmdlLXBhZGRpbmcuaHRt bCA9IFRFWFQKIAorLy8gVGhpcyB0ZXN0IHZlcmlmaWVzIHRoYXQgYSBtaXNtYXRjaCByZWZ0ZXN0 IHdpbGwgZmFpbCBpZiBib3RoIHJlc3VsdHMgYXJlIHNhbWUuCitXT05URklYIDogZmFzdC9oYXJu ZXNzL3NhbXBsZS1mYWlsLW1pc21hdGNoLXJlZnRlc3QuaHRtbCA9IElNQUdFCisKIC8vIC0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tCiAvLyBDSFJPTUUgUkVXUklUVEVOIFRFU1RTCiAvLyAtLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLQo=