59078 2011-04-20 23:34:53 -0700 Reftests should be skipped if pixel tests are disabled. 2011-05-10 20:20:19 -0700 1 1 1 Unclassified WebKit Tools / Tests 528+ (Nightly build) PC OS X 10.5 RESOLVED FIXED P2 Normal --- 36065 1 hayato hayato abarth aroben dpranke eric ojan webkit.review.bot oldest_to_newest 389872 0 hayato 2011-04-20 23:34:53 -0700 We should skip reftests when pixel tests are disabled in new-run-webkit-tests. i.e. a '--no-pixel-tests' is given as a command line option. 389875 1 hayato 2011-04-20 23:38:15 -0700 Maybe, adding a new option for reftests is better than re-using 'pixel-tests' option. 390704 2 90690 hayato 2011-04-22 04:25:26 -0700 Created attachment 90690 skip-reftests-if-pixel-tests-are-disabled 390706 3 hayato 2011-04-22 04:29:05 -0700 Until we have a new option which is specific to reftests, we should respect 'pixel-tests' option whether we execute reftests or not. 393004 4 90690 ojan 2011-04-26 16:27:00 -0700 Comment on attachment 90690 skip-reftests-if-pixel-tests-are-disabled View in context: https://bugs.webkit.org/attachment.cgi?id=90690&action=review > Tools/Scripts/webkitpy/layout_tests/layout_package/test_runner.py:254 > + reftests_files = set(test_file for test_file in self._test_files if self._port.is_reftest(test_file)) > + self._printer.print_expected('%d reftests will be skipped because pixel tests are disabled.' % len(reftests_files)) > + self._test_files -= reftests_files This is a lot of file-system reading just to determine a small number of tests to read. The problem is that we're blocking running any tests on this. I'd rather we not spit out the number of tests skipped and instead skip the test when we're about to run it. 400447 5 92780 hayato 2011-05-09 04:36:56 -0700 Created attachment 92780 update1 400448 6 hayato 2011-05-09 04:38:23 -0700 That's a nice point. Actually, the code took about 6 seconds on my machine if we try to run tests for 'LayoutTest/' directory. This is unacceptable, I guess. I updated the patch so that it skips reftests when we're about to run it. (In reply to comment #4) > (From update of attachment 90690 [details]) > View in context: https://bugs.webkit.org/attachment.cgi?id=90690&action=review > > > Tools/Scripts/webkitpy/layout_tests/layout_package/test_runner.py:254 > > + reftests_files = set(test_file for test_file in self._test_files if self._port.is_reftest(test_file)) > > + self._printer.print_expected('%d reftests will be skipped because pixel tests are disabled.' % len(reftests_files)) > > + self._test_files -= reftests_files > > This is a lot of file-system reading just to determine a small number of tests to read. The problem is that we're blocking running any tests on this. I'd rather we not spit out the number of tests skipped and instead skip the test when we're about to run it. 400522 7 92780 ojan 2011-05-09 08:43:26 -0700 Comment on attachment 92780 update1 View in context: https://bugs.webkit.org/attachment.cgi?id=92780&action=review > Tools/Scripts/webkitpy/layout_tests/layout_package/single_test_runner.py:122 > + # Skips reftests if pixel tests are disabled. This comment isn't very useful. It just describes what the code does. I think it's fine to just delete it. 401001 8 hayato 2011-05-09 19:37:51 -0700 Thank you for the review. I'll land the patch after deleting the comment. (In reply to comment #7) > (From update of attachment 92780 [details]) > View in context: https://bugs.webkit.org/attachment.cgi?id=92780&action=review > > > Tools/Scripts/webkitpy/layout_tests/layout_package/single_test_runner.py:122 > > + # Skips reftests if pixel tests are disabled. > > This comment isn't very useful. It just describes what the code does. I think it's fine to just delete it. 401003 9 hayato 2011-05-09 19:40:53 -0700 Committed r86126: <http://trac.webkit.org/changeset/86126> 401017 10 webkit.review.bot 2011-05-09 20:25:55 -0700 http://trac.webkit.org/changeset/86126 might have broken GTK Linux 32-bit Release 401018 11 eric 2011-05-09 20:34:48 -0700 Why would disabling pixel tests disable ref-tests? Are reftests slow? 401024 12 hayato 2011-05-09 20:51:59 -0700 I agree that it is confusing that the 'pixel tests' option affects 'reftests'. That is not an ideal situation. Let me explain the reason. Reftests are similar to 'pixel tests' in concept. In fact, reftests are implementd in a very similar way to 'pixel tests'. In the current implementation, reftests relies on 'pixel test' mechanism. So disabling pixel tests break reftests. I don't think this is an ideal situation. I appreciate if you could consider this is a temporary solution. We should have a better solution, such as a dedicated a command line option for reftests. (In reply to comment #11) > Why would disabling pixel tests disable ref-tests? Are reftests slow? 401165 13 aroben 2011-05-10 06:38:37 -0700 (In reply to comment #12) > I agree that it is confusing that the 'pixel tests' option affects 'reftests'. > That is not an ideal situation. Let me explain the reason. > > Reftests are similar to 'pixel tests' in concept. In fact, reftests are implementd in a very similar way to 'pixel tests'. > In the current implementation, reftests relies on 'pixel test' mechanism. So disabling pixel tests break reftests. > > I don't think this is an ideal situation. I appreciate if you could consider this is a temporary solution. > We should have a better solution, such as a dedicated a command line option for retests. This reasoning seems a little off to me. Even though pixel tests and reftests are similar from an implementation standpoint, they are not at all similar from a user standpoint. And the user is what matters for command-line options. Pixel tests are typically disabled for some combination of the following reasons: 1) They tend to give a lot of false failures 2) They slow down test runs Presumably (1) does not apply to reftests. (Indeed, that's the whole point!) Does (2) apply, as Eric asked above? If (2) does apply, then it probably makes sense to have a --no-reftests option. Regardless of whether (2) applies, I don't think it makes sense to lump reftests in with pixel tests. They are very different! 401280 14 dpranke 2011-05-10 10:10:12 -0700 I agree with Adam and Eric. The whole point of reftests is that we should be able to use them when we don't want pixel tests but do care about the graphical output, so we shouldn't be disabling them when we disable pixel tests. As to them being slow, logically they'll be a little less than twice as slow as a regular test, but given that we have all of about three of them in the tree, I don't think this is a significant concern at this point. Plus, I'm not a big fan of disabling or skipping tests because they're slow, period. 401286 15 aroben 2011-05-10 10:15:21 -0700 I agree with Dirk. Though I listed slowness as a reason for pixel tests being disabled, I doubt that is the primary reason. (I think false failures is the primary reason.) 401711 16 hayato 2011-05-10 19:20:34 -0700 I totally agree. This patch is a just a quick fix for 'reftest breakage'. I just thought that skipping reftests is better than giving wrong test results until we have a dedicated command line option and fix internal implementations. I'll file a another bug to modify the current implementation so that reftests seems to be independent from 'pixel tests'. (In reply to comment #15) > I agree with Dirk. Though I listed slowness as a reason for pixel tests being disabled, I doubt that is the primary reason. (I think false failures is the primary reason.) (In reply to comment #13) > (In reply to comment #12) > > I agree that it is confusing that the 'pixel tests' option affects 'reftests'. > > That is not an ideal situation. Let me explain the reason. > > > > Reftests are similar to 'pixel tests' in concept. In fact, reftests are implementd in a very similar way to 'pixel tests'. > > In the current implementation, reftests relies on 'pixel test' mechanism. So disabling pixel tests break reftests. > > > > I don't think this is an ideal situation. I appreciate if you could consider this is a temporary solution. > > We should have a better solution, such as a dedicated a command line option for retests. > > This reasoning seems a little off to me. Even though pixel tests and reftests are similar from an implementation standpoint, they are not at all similar from a user standpoint. And the user is what matters for command-line options. > > Pixel tests are typically disabled for some combination of the following reasons: > 1) They tend to give a lot of false failures > 2) They slow down test runs > > Presumably (1) does not apply to reftests. (Indeed, that's the whole point!) Does (2) apply, as Eric asked above? > > If (2) does apply, then it probably makes sense to have a --no-reftests option. Regardless of whether (2) applies, I don't think it makes sense to lump reftests in with pixel tests. They are very different! 401733 17 hayato 2011-05-10 20:20:19 -0700 Filed as https://bugs.webkit.org/show_bug.cgi?id=60605. (In reply to comment #16) > I totally agree. This patch is a just a quick fix for 'reftest breakage'. > I just thought that skipping reftests is better than giving wrong test results until we have a dedicated command line option and fix internal implementations. > > I'll file a another bug to modify the current implementation so that reftests seems to be independent from 'pixel tests'. > > (In reply to comment #15) > > I agree with Dirk. Though I listed slowness as a reason for pixel tests being disabled, I doubt that is the primary reason. (I think false failures is the primary reason.) > > (In reply to comment #13) > > (In reply to comment #12) > > > I agree that it is confusing that the 'pixel tests' option affects 'reftests'. > > > That is not an ideal situation. Let me explain the reason. > > > > > > Reftests are similar to 'pixel tests' in concept. In fact, reftests are implementd in a very similar way to 'pixel tests'. > > > In the current implementation, reftests relies on 'pixel test' mechanism. So disabling pixel tests break reftests. > > > > > > I don't think this is an ideal situation. I appreciate if you could consider this is a temporary solution. > > > We should have a better solution, such as a dedicated a command line option for retests. > > > > This reasoning seems a little off to me. Even though pixel tests and reftests are similar from an implementation standpoint, they are not at all similar from a user standpoint. And the user is what matters for command-line options. > > > > Pixel tests are typically disabled for some combination of the following reasons: > > 1) They tend to give a lot of false failures > > 2) They slow down test runs > > > > Presumably (1) does not apply to reftests. (Indeed, that's the whole point!) Does (2) apply, as Eric asked above? > > > > If (2) does apply, then it probably makes sense to have a --no-reftests option. Regardless of whether (2) applies, I don't think it makes sense to lump reftests in with pixel tests. They are very different! 90690 2011-04-22 04:25:26 -0700 2011-05-09 04:36:46 -0700 skip-reftests-if-pixel-tests-are-disabled bug-59078-20110422202524.patch text/plain 3852 hayato U3VidmVyc2lvbiBSZXZpc2lvbjogODQzNTAKZGlmZiAtLWdpdCBhL1Rvb2xzL0NoYW5nZUxvZyBi L1Rvb2xzL0NoYW5nZUxvZwppbmRleCA4ZDFhMWFhM2ZiM2MwOTQwM2Y2NDgxYWM1MTg0Yzg0OGU3 NWI1MGQ5Li42YWJlNTJiNTc5ZDBhNjQ4NzM1ZjU3YjFiMjJlZmFhNDAyNGE2MDk2IDEwMDY0NAot LS0gYS9Ub29scy9DaGFuZ2VMb2cKKysrIGIvVG9vbHMvQ2hhbmdlTG9nCkBAIC0xLDMgKzEsMTUg QEAKKzIwMTEtMDQtMjIgIEhheWF0byBJdG8gIDxoYXlhdG9AY2hyb21pdW0ub3JnPgorCisgICAg ICAgIFJldmlld2VkIGJ5IE5PQk9EWSAoT09QUyEpLgorCisgICAgICAgIFNraXAgcmVmdGVzdHMg aWYgcGl4ZWwgdGVzdHMgYXJlIGRpc2FibGVkLgorCisgICAgICAgIGh0dHBzOi8vYnVncy53ZWJr aXQub3JnL3Nob3dfYnVnLmNnaT9pZD01OTA3OAorCisgICAgICAgICogU2NyaXB0cy93ZWJraXRw eS9sYXlvdXRfdGVzdHMvbGF5b3V0X3BhY2thZ2UvdGVzdF9ydW5uZXIucHk6CisgICAgICAgICog U2NyaXB0cy93ZWJraXRweS9sYXlvdXRfdGVzdHMvcG9ydC9iYXNlLnB5OgorICAgICAgICAqIFNj cmlwdHMvd2Via2l0cHkvbGF5b3V0X3Rlc3RzL3J1bl93ZWJraXRfdGVzdHNfdW5pdHRlc3QucHk6 CisKIDIwMTEtMDQtMTkgIEVyaWMgU2VpZGVsICA8ZXJpY0B3ZWJraXQub3JnPgogCiAgICAgICAg IFJldmlld2VkIGJ5IEFkYW0gQmFydGguCmRpZmYgLS1naXQgYS9Ub29scy9TY3JpcHRzL3dlYmtp dHB5L2xheW91dF90ZXN0cy9sYXlvdXRfcGFja2FnZS90ZXN0X3J1bm5lci5weSBiL1Rvb2xzL1Nj cmlwdHMvd2Via2l0cHkvbGF5b3V0X3Rlc3RzL2xheW91dF9wYWNrYWdlL3Rlc3RfcnVubmVyLnB5 CmluZGV4IDhlNTM0YjE1ZjU1Y2FhYjhmNDEyODI3ODQ1YzA3ODAzMTM1ZWE1MTYuLmU1Y2UzNmZl OWIzNGQxNzRhOGU1MGEzM2NlZWQwZGJlNGM1YjY0NWYgMTAwNjQ0Ci0tLSBhL1Rvb2xzL1Njcmlw dHMvd2Via2l0cHkvbGF5b3V0X3Rlc3RzL2xheW91dF9wYWNrYWdlL3Rlc3RfcnVubmVyLnB5Cisr KyBiL1Rvb2xzL1NjcmlwdHMvd2Via2l0cHkvbGF5b3V0X3Rlc3RzL2xheW91dF9wYWNrYWdlL3Rl c3RfcnVubmVyLnB5CkBAIC0yNDcsNiArMjQ3LDExIEBAIGNsYXNzIFRlc3RSdW5uZXI6CiAgICAg ICAgIGlmIHNlbGYuX29wdGlvbnMudGVzdF9saXN0OgogICAgICAgICAgICAgcGF0aHMgKz0gc2Vs Zi5fc3RyaXBfdGVzdF9kaXJfcHJlZml4ZXMocmVhZF90ZXN0X2ZpbGVzKHNlbGYuX2ZzLCBzZWxm Ll9vcHRpb25zLnRlc3RfbGlzdCkpCiAgICAgICAgIHNlbGYuX3Rlc3RfZmlsZXMgPSBzZWxmLl9w b3J0LnRlc3RzKHBhdGhzKQorICAgICAgICBpZiBub3Qgc2VsZi5fcG9ydC5nZXRfb3B0aW9uKCdw aXhlbF90ZXN0cycpOgorICAgICAgICAgICAgIyBTa2lwIHJlZnRlc3RzIGlmIHBpeGVsIHRlc3Rz IGFyZSBkaXNhYmxlZC4KKyAgICAgICAgICAgIHJlZnRlc3RzX2ZpbGVzID0gc2V0KHRlc3RfZmls ZSBmb3IgdGVzdF9maWxlIGluIHNlbGYuX3Rlc3RfZmlsZXMgaWYgc2VsZi5fcG9ydC5pc19yZWZ0 ZXN0KHRlc3RfZmlsZSkpCisgICAgICAgICAgICBzZWxmLl9wcmludGVyLnByaW50X2V4cGVjdGVk KCclZCByZWZ0ZXN0cyB3aWxsIGJlIHNraXBwZWQgYmVjYXVzZSBwaXhlbCB0ZXN0cyBhcmUgZGlz YWJsZWQuJyAlIGxlbihyZWZ0ZXN0c19maWxlcykpCisgICAgICAgICAgICBzZWxmLl90ZXN0X2Zp bGVzIC09IHJlZnRlc3RzX2ZpbGVzCiAKICAgICBkZWYgX3N0cmlwX3Rlc3RfZGlyX3ByZWZpeGVz KHNlbGYsIHBhdGhzKToKICAgICAgICAgcmV0dXJuIFtzZWxmLl9zdHJpcF90ZXN0X2Rpcl9wcmVm aXgocGF0aCkgZm9yIHBhdGggaW4gcGF0aHMgaWYgcGF0aF0KZGlmZiAtLWdpdCBhL1Rvb2xzL1Nj cmlwdHMvd2Via2l0cHkvbGF5b3V0X3Rlc3RzL3BvcnQvYmFzZS5weSBiL1Rvb2xzL1NjcmlwdHMv d2Via2l0cHkvbGF5b3V0X3Rlc3RzL3BvcnQvYmFzZS5weQppbmRleCBjMmU1NjVlYTE1MjU0NzUx ZDJlMmRiYzBlZDQ2ZjY4NDhlYzkzNTQ0Li5iNGNiMmYxN2YxMmI5ZmJmNzkwYzI0ODYwMmViNDk4 YTE0NzljOTE2IDEwMDc1NQotLS0gYS9Ub29scy9TY3JpcHRzL3dlYmtpdHB5L2xheW91dF90ZXN0 cy9wb3J0L2Jhc2UucHkKKysrIGIvVG9vbHMvU2NyaXB0cy93ZWJraXRweS9sYXlvdXRfdGVzdHMv cG9ydC9iYXNlLnB5CkBAIC0zNzksNiArMzc5LDEyIEBAIGNsYXNzIFBvcnQob2JqZWN0KToKICAg ICAgICAgdGV4dCA9IHNlbGYuX2ZpbGVzeXN0ZW0ucmVhZF9iaW5hcnlfZmlsZShwYXRoKQogICAg ICAgICByZXR1cm4gdGV4dC5yZXBsYWNlKCJcclxuIiwgIlxuIikKIAorICAgIGRlZiBpc19yZWZ0 ZXN0KHNlbGYsIGZpbGVuYW1lKToKKyAgICAgICAgIiIiUmV0dXJuIHRydWUgaWYgdGhlIGZpbGVu YW1lIHBvaW50cyB0byBhIHJlZnRlc3QuIiIiCisgICAgICAgIGZzID0gc2VsZi5fZmlsZXN5c3Rl bQorICAgICAgICByZXR1cm4gKGZzLmV4aXN0cyhzZWxmLnJlZnRlc3RfZXhwZWN0ZWRfZmlsZW5h bWUoZmlsZW5hbWUpKSBvcgorICAgICAgICAgICAgICAgIGZzLmV4aXN0cyhzZWxmLnJlZnRlc3Rf ZXhwZWN0ZWRfbWlzbWF0Y2hfZmlsZW5hbWUoZmlsZW5hbWUpKSkKKwogICAgIGRlZiByZWZ0ZXN0 X2V4cGVjdGVkX2ZpbGVuYW1lKHNlbGYsIGZpbGVuYW1lKToKICAgICAgICAgIiIiUmV0dXJuIHRo ZSBmaWxlbmFtZSBvZiByZWZlcmVuY2Ugd2UgZXhwZWN0IHRoZSB0ZXN0IG1hdGNoZXMuIiIiCiAg ICAgICAgIHJldHVybiBzZWxmLmV4cGVjdGVkX2ZpbGVuYW1lKGZpbGVuYW1lLCAnLmh0bWwnKQpk aWZmIC0tZ2l0IGEvVG9vbHMvU2NyaXB0cy93ZWJraXRweS9sYXlvdXRfdGVzdHMvcnVuX3dlYmtp dF90ZXN0c191bml0dGVzdC5weSBiL1Rvb2xzL1NjcmlwdHMvd2Via2l0cHkvbGF5b3V0X3Rlc3Rz L3J1bl93ZWJraXRfdGVzdHNfdW5pdHRlc3QucHkKaW5kZXggOTQwYjRiODc5ZWQ0YmU0MTNkYjI0 NTRmZmQyYjQ1ZmQyZjllMGVhOC4uZTIyMGY1N2IzN2E0YzMxZDcyZWM1N2ZiYTg4MDQyYjk5N2Vl ZWJlYSAxMDA2NDQKLS0tIGEvVG9vbHMvU2NyaXB0cy93ZWJraXRweS9sYXlvdXRfdGVzdHMvcnVu X3dlYmtpdF90ZXN0c191bml0dGVzdC5weQorKysgYi9Ub29scy9TY3JpcHRzL3dlYmtpdHB5L2xh eW91dF90ZXN0cy9ydW5fd2Via2l0X3Rlc3RzX3VuaXR0ZXN0LnB5CkBAIC01NTksNiArNTU5LDEw IEBAIGNsYXNzIE1haW5UZXN0KHVuaXR0ZXN0LlRlc3RDYXNlKToKICAgICAgICAgdGVzdHNfcnVu ID0gZ2V0X3Rlc3RzX3J1bihbJ3Bhc3Nlcy9yZWZ0ZXN0Lmh0bWwnXSwgdGVzdHNfaW5jbHVkZWQ9 VHJ1ZSwgZmxhdHRlbl9iYXRjaGVzPVRydWUpCiAgICAgICAgIHNlbGYuYXNzZXJ0RXF1YWxzKFsn cGFzc2VzL3JlZnRlc3QuaHRtbCddLCB0ZXN0c19ydW4pCiAKKyAgICBkZWYgdGVzdF9yZWZ0ZXN0 X3NraXBfcmVmdGVzdHNfaWZfcGl4ZWxfdGVzdHNfYXJlX2Rpc2FibGVkKHNlbGYpOgorICAgICAg ICB0ZXN0c19ydW4gPSBnZXRfdGVzdHNfcnVuKFsnLS1uby1waXhlbC10ZXN0cycsICdwYXNzZXMv cmVmdGVzdC5odG1sJ10sIHRlc3RzX2luY2x1ZGVkPVRydWUsIGZsYXR0ZW5fYmF0Y2hlcz1UcnVl KQorICAgICAgICBzZWxmLmFzc2VydEVxdWFscyhbXSwgdGVzdHNfcnVuKQorCiAgICAgZGVmIHRl c3RfcmVmdGVzdF9leHBlY3RlZF9odG1sX3Nob3VsZF9iZV9pZ25vcmVkKHNlbGYpOgogICAgICAg ICB0ZXN0c19ydW4gPSBnZXRfdGVzdHNfcnVuKFsncGFzc2VzL3JlZnRlc3QtZXhwZWN0ZWQuaHRt bCddLCB0ZXN0c19pbmNsdWRlZD1UcnVlLCBmbGF0dGVuX2JhdGNoZXM9VHJ1ZSkKICAgICAgICAg c2VsZi5hc3NlcnRFcXVhbHMoW10sIHRlc3RzX3J1bikK 92780 2011-05-09 04:36:56 -0700 2011-05-09 08:43:26 -0700 update1 bug-59078-20110509203654.patch text/plain 6274 hayato U3VidmVyc2lvbiBSZXZpc2lvbjogODYwMzgKZGlmZiAtLWdpdCBhL1Rvb2xzL0NoYW5nZUxvZyBi L1Rvb2xzL0NoYW5nZUxvZwppbmRleCAzMDA1YmQyYmY5N2ViYzYzYWZjNTVkMzI4OTBjNmRmM2Fl NGE3NjU3Li41NzU4NDg0NDIyMGFiM2M3OGFkZDE3M2NhNmQ2ZDQyMDBjMGEzYTBkIDEwMDY0NAot LS0gYS9Ub29scy9DaGFuZ2VMb2cKKysrIGIvVG9vbHMvQ2hhbmdlTG9nCkBAIC0xLDMgKzEsMTYg QEAKKzIwMTEtMDUtMDkgIEhheWF0byBJdG8gIDxoYXlhdG9AY2hyb21pdW0ub3JnPgorCisgICAg ICAgIFJldmlld2VkIGJ5IE5PQk9EWSAoT09QUyEpLgorCisgICAgICAgIFNraXAgcmVmdGVzdHMg aWYgcGl4ZWwgdGVzdHMgYXJlIGRpc2FibGVkLgorCisgICAgICAgIGh0dHBzOi8vYnVncy53ZWJr aXQub3JnL3Nob3dfYnVnLmNnaT9pZD01OTA3OAorCisgICAgICAgICogU2NyaXB0cy93ZWJraXRw eS9sYXlvdXRfdGVzdHMvbGF5b3V0X3BhY2thZ2UvbWFuYWdlci5weToKKyAgICAgICAgKiBTY3Jp cHRzL3dlYmtpdHB5L2xheW91dF90ZXN0cy9sYXlvdXRfcGFja2FnZS9zaW5nbGVfdGVzdF9ydW5u ZXIucHk6CisgICAgICAgICogU2NyaXB0cy93ZWJraXRweS9sYXlvdXRfdGVzdHMvbGF5b3V0X3Bh Y2thZ2Uvd29ya2VyLnB5OgorICAgICAgICAqIFNjcmlwdHMvd2Via2l0cHkvbGF5b3V0X3Rlc3Rz L3J1bl93ZWJraXRfdGVzdHNfaW50ZWdyYXRpb250ZXN0LnB5OgorCiAyMDExLTA1LTA3ICBBZGFt IEJhcnRoICA8YWJhcnRoQHdlYmtpdC5vcmc+CiAKICAgICAgICAgUmV2aWV3ZWQgYnkgRXJpYyBT ZWlkZWwuCmRpZmYgLS1naXQgYS9Ub29scy9TY3JpcHRzL3dlYmtpdHB5L2xheW91dF90ZXN0cy9s YXlvdXRfcGFja2FnZS9tYW5hZ2VyLnB5IGIvVG9vbHMvU2NyaXB0cy93ZWJraXRweS9sYXlvdXRf dGVzdHMvbGF5b3V0X3BhY2thZ2UvbWFuYWdlci5weQppbmRleCBjMjBmOGE0MzgwMzJjMzdkYjAy NTA4OTcyNDllN2ZmNDEyYjVhMTk5Li43NGI4ZjgwZjAzZTMzODJkOTBjZmQ1MGY5YjU2MmQ4YWZi YjQyMWM3IDEwMDY0NAotLS0gYS9Ub29scy9TY3JpcHRzL3dlYmtpdHB5L2xheW91dF90ZXN0cy9s YXlvdXRfcGFja2FnZS9tYW5hZ2VyLnB5CisrKyBiL1Rvb2xzL1NjcmlwdHMvd2Via2l0cHkvbGF5 b3V0X3Rlc3RzL2xheW91dF9wYWNrYWdlL21hbmFnZXIucHkKQEAgLTg2MCwxNSArODYwLDE4IEBA IGNsYXNzIE1hbmFnZXI6CiAgICAgICAgICAgICBzZWxmLl91cGRhdGVfc3VtbWFyeV93aXRoX3Jl c3VsdChyZXN1bHRfc3VtbWFyeSwgcmVzdWx0KQogCiAgICAgZGVmIF91cGRhdGVfc3VtbWFyeV93 aXRoX3Jlc3VsdChzZWxmLCByZXN1bHRfc3VtbWFyeSwgcmVzdWx0KToKLSAgICAgICAgZXhwZWN0 ZWQgPSBzZWxmLl9leHBlY3RhdGlvbnMubWF0Y2hlc19hbl9leHBlY3RlZF9yZXN1bHQoCi0gICAg ICAgICAgICByZXN1bHQuZmlsZW5hbWUsIHJlc3VsdC50eXBlLCBzZWxmLl9vcHRpb25zLnBpeGVs X3Rlc3RzKQotICAgICAgICByZXN1bHRfc3VtbWFyeS5hZGQocmVzdWx0LCBleHBlY3RlZCkKLSAg ICAgICAgZXhwX3N0ciA9IHNlbGYuX2V4cGVjdGF0aW9ucy5nZXRfZXhwZWN0YXRpb25zX3N0cmlu ZygKLSAgICAgICAgICAgIHJlc3VsdC5maWxlbmFtZSkKLSAgICAgICAgZ290X3N0ciA9IHNlbGYu X2V4cGVjdGF0aW9ucy5leHBlY3RhdGlvbl90b19zdHJpbmcocmVzdWx0LnR5cGUpCi0gICAgICAg IHNlbGYuX3ByaW50ZXIucHJpbnRfdGVzdF9yZXN1bHQocmVzdWx0LCBleHBlY3RlZCwgZXhwX3N0 ciwgZ290X3N0cikKKyAgICAgICAgaWYgcmVzdWx0LnR5cGUgPT0gdGVzdF9leHBlY3RhdGlvbnMu U0tJUDoKKyAgICAgICAgICAgIHJlc3VsdF9zdW1tYXJ5LmFkZChyZXN1bHQsIGV4cGVjdGVkPVRy dWUpCisgICAgICAgIGVsc2U6CisgICAgICAgICAgICBleHBlY3RlZCA9IHNlbGYuX2V4cGVjdGF0 aW9ucy5tYXRjaGVzX2FuX2V4cGVjdGVkX3Jlc3VsdCgKKyAgICAgICAgICAgICAgICByZXN1bHQu ZmlsZW5hbWUsIHJlc3VsdC50eXBlLCBzZWxmLl9vcHRpb25zLnBpeGVsX3Rlc3RzKQorICAgICAg ICAgICAgcmVzdWx0X3N1bW1hcnkuYWRkKHJlc3VsdCwgZXhwZWN0ZWQpCisgICAgICAgICAgICBl eHBfc3RyID0gc2VsZi5fZXhwZWN0YXRpb25zLmdldF9leHBlY3RhdGlvbnNfc3RyaW5nKAorICAg ICAgICAgICAgICAgIHJlc3VsdC5maWxlbmFtZSkKKyAgICAgICAgICAgIGdvdF9zdHIgPSBzZWxm Ll9leHBlY3RhdGlvbnMuZXhwZWN0YXRpb25fdG9fc3RyaW5nKHJlc3VsdC50eXBlKQorICAgICAg ICAgICAgc2VsZi5fcHJpbnRlci5wcmludF90ZXN0X3Jlc3VsdChyZXN1bHQsIGV4cGVjdGVkLCBl eHBfc3RyLCBnb3Rfc3RyKQogICAgICAgICBzZWxmLl9wcmludGVyLnByaW50X3Byb2dyZXNzKHJl c3VsdF9zdW1tYXJ5LCBzZWxmLl9yZXRyeWluZywKLSAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICBzZWxmLl90ZXN0X2ZpbGVzX2xpc3QpCisgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgc2VsZi5fdGVzdF9maWxlc19saXN0KQogCiAgICAgICAgIGRlZiBp bnRlcnJ1cHRfaWZfYXRfZmFpbHVyZV9saW1pdChsaW1pdCwgY291bnQsIG1lc3NhZ2UpOgogICAg ICAgICAgICAgaWYgbGltaXQgYW5kIGNvdW50ID49IGxpbWl0OgpkaWZmIC0tZ2l0IGEvVG9vbHMv U2NyaXB0cy93ZWJraXRweS9sYXlvdXRfdGVzdHMvbGF5b3V0X3BhY2thZ2Uvc2luZ2xlX3Rlc3Rf cnVubmVyLnB5IGIvVG9vbHMvU2NyaXB0cy93ZWJraXRweS9sYXlvdXRfdGVzdHMvbGF5b3V0X3Bh Y2thZ2Uvc2luZ2xlX3Rlc3RfcnVubmVyLnB5CmluZGV4IGZhYjdmZTVlZmRiMGFhYjYxNTkwZjc5 Yjc3ZmQ3ZjJhMDk5YTUzM2IuLjhkOTU2OTJiOTZiYTliMjQ1NGFjNWVjNzU5OGI3YmY2OTI4OWUw OWUgMTAwNjQ0Ci0tLSBhL1Rvb2xzL1NjcmlwdHMvd2Via2l0cHkvbGF5b3V0X3Rlc3RzL2xheW91 dF9wYWNrYWdlL3NpbmdsZV90ZXN0X3J1bm5lci5weQorKysgYi9Ub29scy9TY3JpcHRzL3dlYmtp dHB5L2xheW91dF90ZXN0cy9sYXlvdXRfcGFja2FnZS9zaW5nbGVfdGVzdF9ydW5uZXIucHkKQEAg LTMxLDYgKzMxLDcgQEAgaW1wb3J0IGxvZ2dpbmcKIGltcG9ydCB0aW1lCiAKIGZyb20gd2Via2l0 cHkubGF5b3V0X3Rlc3RzLnBvcnQgaW1wb3J0IGJhc2UKK2Zyb20gd2Via2l0cHkubGF5b3V0X3Rl c3RzLmxheW91dF9wYWNrYWdlIGltcG9ydCB0ZXN0X2V4cGVjdGF0aW9ucwogZnJvbSB3ZWJraXRw eS5sYXlvdXRfdGVzdHMubGF5b3V0X3BhY2thZ2UgaW1wb3J0IHRlc3RfZmFpbHVyZXMKIGZyb20g d2Via2l0cHkubGF5b3V0X3Rlc3RzLmxheW91dF9wYWNrYWdlIGltcG9ydCB0ZXN0X3Jlc3VsdF93 cml0ZXIKIGZyb20gd2Via2l0cHkubGF5b3V0X3Rlc3RzLmxheW91dF9wYWNrYWdlLnRlc3RfcmVz dWx0cyBpbXBvcnQgVGVzdFJlc3VsdApAQCAtMTE2LDcgKzExNywxMiBAQCBjbGFzcyBTaW5nbGVU ZXN0UnVubmVyOgogICAgICAgICAgICAgZWxzZToKICAgICAgICAgICAgICAgICByZXR1cm4gc2Vs Zi5fcnVuX3JlYmFzZWxpbmUoKQogICAgICAgICBpZiBzZWxmLl9pc19yZWZ0ZXN0OgotICAgICAg ICAgICAgcmV0dXJuIHNlbGYuX3J1bl9yZWZ0ZXN0KCkKKyAgICAgICAgICAgIGlmIHNlbGYuX3Bv cnQuZ2V0X29wdGlvbigncGl4ZWxfdGVzdHMnKToKKyAgICAgICAgICAgICAgICByZXR1cm4gc2Vs Zi5fcnVuX3JlZnRlc3QoKQorICAgICAgICAgICAgIyBTa2lwcyByZWZ0ZXN0cyBpZiBwaXhlbCB0 ZXN0cyBhcmUgZGlzYWJsZWQuCisgICAgICAgICAgICByZXN1bHQgPSBUZXN0UmVzdWx0KHNlbGYu X2ZpbGVuYW1lKQorICAgICAgICAgICAgcmVzdWx0LnR5cGUgPSB0ZXN0X2V4cGVjdGF0aW9ucy5T S0lQCisgICAgICAgICAgICByZXR1cm4gcmVzdWx0CiAgICAgICAgIHJldHVybiBzZWxmLl9ydW5f Y29tcGFyZV90ZXN0KCkKIAogICAgIGRlZiBfcnVuX2NvbXBhcmVfdGVzdChzZWxmKToKZGlmZiAt LWdpdCBhL1Rvb2xzL1NjcmlwdHMvd2Via2l0cHkvbGF5b3V0X3Rlc3RzL2xheW91dF9wYWNrYWdl L3dvcmtlci5weSBiL1Rvb2xzL1NjcmlwdHMvd2Via2l0cHkvbGF5b3V0X3Rlc3RzL2xheW91dF9w YWNrYWdlL3dvcmtlci5weQppbmRleCBkMjk0MDM1YTZlNWY5ZDE1OTIyNTc3NWY2ZjU1MjQwNTk4 YzEwNjM2Li42ZmYzZjEyNThhMjBlNzk3YjUxMmQ2ZjUyMTllOTk2ZWJlNDNmNzhkIDEwMDY0NAot LS0gYS9Ub29scy9TY3JpcHRzL3dlYmtpdHB5L2xheW91dF90ZXN0cy9sYXlvdXRfcGFja2FnZS93 b3JrZXIucHkKKysrIGIvVG9vbHMvU2NyaXB0cy93ZWJraXRweS9sYXlvdXRfdGVzdHMvbGF5b3V0 X3BhY2thZ2Uvd29ya2VyLnB5CkBAIC0zNiw2ICszNiw3IEBAIGltcG9ydCB0aW1lCiBmcm9tIHdl YmtpdHB5LmNvbW1vbi5zeXN0ZW0gaW1wb3J0IHN0YWNrX3V0aWxzCiBmcm9tIHdlYmtpdHB5Lmxh eW91dF90ZXN0cy5sYXlvdXRfcGFja2FnZSBpbXBvcnQgbWFuYWdlcl93b3JrZXJfYnJva2VyCiBm cm9tIHdlYmtpdHB5LmxheW91dF90ZXN0cy5sYXlvdXRfcGFja2FnZSBpbXBvcnQgc2luZ2xlX3Rl c3RfcnVubmVyCitmcm9tIHdlYmtpdHB5LmxheW91dF90ZXN0cy5sYXlvdXRfcGFja2FnZSBpbXBv cnQgdGVzdF9leHBlY3RhdGlvbnMKIGZyb20gd2Via2l0cHkubGF5b3V0X3Rlc3RzLmxheW91dF9w YWNrYWdlIGltcG9ydCB0ZXN0X3Jlc3VsdHMKIAogX2xvZyA9IGxvZ2dpbmcuZ2V0TG9nZ2VyKF9f bmFtZV9fKQpAQCAtMjE0LDYgKzIxNSw4IEBAIGNsYXNzIFdvcmtlcihtYW5hZ2VyX3dvcmtlcl9i cm9rZXIuQWJzdHJhY3RXb3JrZXIpOgogICAgICAgICAgICAgX2xvZy5kZWJ1ZygiJXMgJXMgZmFp bGVkOiIgJSAoc2VsZi5fbmFtZSwgdGVzdF9uYW1lKSkKICAgICAgICAgICAgIGZvciBmIGluIHJl c3VsdC5mYWlsdXJlczoKICAgICAgICAgICAgICAgICBfbG9nLmRlYnVnKCIlcyAgJXMiICUgKHNl bGYuX25hbWUsIGYubWVzc2FnZSgpKSkKKyAgICAgICAgZWxpZiByZXN1bHQudHlwZSA9PSB0ZXN0 X2V4cGVjdGF0aW9ucy5TS0lQOgorICAgICAgICAgICAgX2xvZy5kZWJ1ZygiJXMgJXMgc2tpcHBl ZCIgJSAoc2VsZi5fbmFtZSwgdGVzdF9uYW1lKSkKICAgICAgICAgZWxzZToKICAgICAgICAgICAg IF9sb2cuZGVidWcoIiVzICVzIHBhc3NlZCIgJSAoc2VsZi5fbmFtZSwgdGVzdF9uYW1lKSkKIApk aWZmIC0tZ2l0IGEvVG9vbHMvU2NyaXB0cy93ZWJraXRweS9sYXlvdXRfdGVzdHMvcnVuX3dlYmtp dF90ZXN0c19pbnRlZ3JhdGlvbnRlc3QucHkgYi9Ub29scy9TY3JpcHRzL3dlYmtpdHB5L2xheW91 dF90ZXN0cy9ydW5fd2Via2l0X3Rlc3RzX2ludGVncmF0aW9udGVzdC5weQppbmRleCAxMzFlM2Y0 ZDk5NDA2MjcwNzQ2MThmNTllOGQzMWVjNzU1OTdlOWYzLi5mMjMyY2EyOTFkMWFhNmJhZmQ2ODJl OGZmMTJlYTEzYTVlMzZlM2Q0IDEwMDY0NAotLS0gYS9Ub29scy9TY3JpcHRzL3dlYmtpdHB5L2xh eW91dF90ZXN0cy9ydW5fd2Via2l0X3Rlc3RzX2ludGVncmF0aW9udGVzdC5weQorKysgYi9Ub29s cy9TY3JpcHRzL3dlYmtpdHB5L2xheW91dF90ZXN0cy9ydW5fd2Via2l0X3Rlc3RzX2ludGVncmF0 aW9udGVzdC5weQpAQCAtNTc3LDYgKzU3NywxMCBAQCBjbGFzcyBNYWluVGVzdCh1bml0dGVzdC5U ZXN0Q2FzZSk6CiAgICAgICAgIHRlc3RzX3J1biA9IGdldF90ZXN0c19ydW4oWydwYXNzZXMvcmVm dGVzdC5odG1sJ10sIHRlc3RzX2luY2x1ZGVkPVRydWUsIGZsYXR0ZW5fYmF0Y2hlcz1UcnVlKQog ICAgICAgICBzZWxmLmFzc2VydEVxdWFscyhbJ3Bhc3Nlcy9yZWZ0ZXN0Lmh0bWwnXSwgdGVzdHNf cnVuKQogCisgICAgZGVmIHRlc3RfcmVmdGVzdF9za2lwX3JlZnRlc3RzX2lmX3BpeGVsX3Rlc3Rz X2FyZV9kaXNhYmxlZChzZWxmKToKKyAgICAgICAgdGVzdHNfcnVuID0gZ2V0X3Rlc3RzX3J1bihb Jy0tbm8tcGl4ZWwtdGVzdHMnLCAncGFzc2VzL3JlZnRlc3QuaHRtbCddLCB0ZXN0c19pbmNsdWRl ZD1UcnVlLCBmbGF0dGVuX2JhdGNoZXM9VHJ1ZSkKKyAgICAgICAgc2VsZi5hc3NlcnRFcXVhbHMo W10sIHRlc3RzX3J1bikKKwogICAgIGRlZiB0ZXN0X3JlZnRlc3RfZXhwZWN0ZWRfaHRtbF9zaG91 bGRfYmVfaWdub3JlZChzZWxmKToKICAgICAgICAgdGVzdHNfcnVuID0gZ2V0X3Rlc3RzX3J1bihb J3Bhc3Nlcy9yZWZ0ZXN0LWV4cGVjdGVkLmh0bWwnXSwgdGVzdHNfaW5jbHVkZWQ9VHJ1ZSwgZmxh dHRlbl9iYXRjaGVzPVRydWUpCiAgICAgICAgIHNlbGYuYXNzZXJ0RXF1YWxzKFtdLCB0ZXN0c19y dW4pCg==