76764 2012-01-20 18:33:21 -0800 run-perf-tests ignore Skipped list on chromium 2012-01-23 14:46:30 -0800 1 1 1 Unclassified WebKit Tools / Tests 528+ (Nightly build) Unspecified Unspecified RESOLVED FIXED P2 Normal --- 10266 1 rniwa rniwa abarth dpranke eric ojan webkit.review.bot oldest_to_newest 539993 0 rniwa 2012-01-20 18:33:21 -0800 Apparently Chromium port doesn't inherit from Base port :( 539994 1 rniwa 2012-01-20 18:33:36 -0800 (In reply to comment #0) > Apparently Chromium port doesn't inherit from Base port :( I mean WebKit* port. 539997 2 eric 2012-01-20 18:38:00 -0800 Yes. Base should die. Most of its logic should split off into other objects, and what remains should be folded up into WebKit. Chromium needs to become a real boy some day. 540000 3 rniwa 2012-01-20 18:41:40 -0800 (In reply to comment #2) > Yes. Base should die. Most of its logic should split off into other objects, and what remains should be folded up into WebKit. Chromium needs to become a real boy some day. That sounds like a good idea. But, I think I'm going to take the easy way out and move stuff from WebKit to Base for now :( Making Chromium a real WebKit port seems to require too much effort just to fix this bug. 540001 4 dpranke 2012-01-20 18:47:19 -0800 I agree that BasePort and WebKitPort should probably merge, but merging webkit and chromium is a fair pile of work (that I'm gradually working on) and we need to do that carefully. Not something to tackle here :). 540341 5 123505 rniwa 2012-01-22 17:59:44 -0800 Created attachment 123505 fixes the bug 540431 6 123505 abarth 2012-01-23 01:28:34 -0800 Comment on attachment 123505 fixes the bug View in context: https://bugs.webkit.org/attachment.cgi?id=123505&action=review > Tools/ChangeLog:10 > + layout tests but performacne tests don't use test_expectations.txt so Chromium port performacne: typo 540432 7 abarth 2012-01-23 01:29:29 -0800 I understand that you're just moving code around, but this stuff is poorly factored. The base port shouldn't be a god object that understands details of how to parse skipped lists. 540605 8 rniwa 2012-01-23 09:27:20 -0800 (In reply to comment #7) > I understand that you're just moving code around, but this stuff is poorly factored. The base port shouldn't be a god object that understands details of how to parse skipped lists. It's not obvious what the correct refactoring is. It seems that the first refactoring we need to do is to turn Chromium into a proper webkit port. 540713 9 abarth 2012-01-23 11:47:40 -0800 > It's not obvious what the correct refactoring is. It seems that the first refactoring we need to do is to turn Chromium into a proper webkit port. I more incremental approach is to create a class for Skipped lists that knows how to parser them and translate them into expectations rather than have the port know all those details. 540742 10 rniwa 2012-01-23 12:27:54 -0800 (In reply to comment #9) > > It's not obvious what the correct refactoring is. It seems that the first refactoring we need to do is to turn Chromium into a proper webkit port. > > I more incremental approach is to create a class for Skipped lists that knows how to parser them and translate them into expectations rather than have the port know all those details. Do you strongly feel that it's a blocker for this bug? If not, I'm not motivated to do that the refactoring. 540796 11 123505 dpranke 2012-01-23 13:41:36 -0800 Comment on attachment 123505 fixes the bug I think this change is fine. While I agree with the general "god" sentiment about the Port class, it's not at all clear what the right way to factor out logic is here, and creating a class just for parsing Skipped files itself would be silly. I think perhaps a separate change that moved all of the expectations-related logic out might be good, but I don't really want to make that a blocker of this and I'd want to think about that more. 540802 12 123505 dpranke 2012-01-23 13:43:32 -0800 Comment on attachment 123505 fixes the bug View in context: https://bugs.webkit.org/attachment.cgi?id=123505&action=review > Tools/Scripts/webkitpy/layout_tests/port/base.py:599 > + @memoized Note that I wouldn't bother with the @memoized here (I realize that the WebKitPort had it); this routine can't be called all that often, and memoizing it but not other expectations-related routines might lead to weird consistency issues. 540803 13 rniwa 2012-01-23 13:43:58 -0800 (In reply to comment #11) > I think perhaps a separate change that moved all of the expectations-related logic out might be good, but I don't really want to make that a blocker of this and I'd want to think about that more. That sounds like a good idea. Maybe we can create a class for the "list of tests" that can parse test_expectations.txt and Skipped. 540804 14 rniwa 2012-01-23 13:44:43 -0800 (In reply to comment #12) > (From update of attachment 123505 [details]) > View in context: https://bugs.webkit.org/attachment.cgi?id=123505&action=review > > > Tools/Scripts/webkitpy/layout_tests/port/base.py:599 > > + @memoized > > Note that I wouldn't bother with the @memoized here (I realize that the WebKitPort had it); this routine can't be called all that often, and memoizing it but not other expectations-related routines might lead to weird consistency issues. This function is called by skips_perf_test, which is called per test. So it should be memoized. 540825 15 dpranke 2012-01-23 14:09:25 -0800 (In reply to comment #14) > (In reply to comment #12) > > (From update of attachment 123505 [details] [details]) > > View in context: https://bugs.webkit.org/attachment.cgi?id=123505&action=review > > > > > Tools/Scripts/webkitpy/layout_tests/port/base.py:599 > > > + @memoized > > > > Note that I wouldn't bother with the @memoized here (I realize that the WebKitPort had it); this routine can't be called all that often, and memoizing it but not other expectations-related routines might lead to weird consistency issues. > > This function is called by skips_perf_test, which is called per test. So it should be memoized. Okay, for now :). Ultimately I would probably put skips_perf_test() in perftestrunner and just merged it into _collect_tests(). Neither skips_layout_test or skips_perf_test() really belongs on the Port object (see bug 47528). The port object should have no real knowledge of the actual expectations, just a way to provide the contents of the files. Memoization often (not always) suggests to me that you have the wrong algorithmic design and you should fix the algorithm rather than relying on memoizing to get the performance you need. 540839 16 rniwa 2012-01-23 14:31:20 -0800 (In reply to comment #15) > Ultimately I would probably put skips_perf_test() in perftestrunner and just merged it into _collect_tests(). Neither skips_layout_test or skips_perf_test() really belongs on the Port object (see bug 47528). The port object should have no real knowledge of the actual expectations, just a way to provide the contents of the files. Memoization often (not always) suggests to me that you have the wrong algorithmic design and you should fix the algorithm rather than relying on memoizing to get the performance you need. That makes a lot of sense. I added perf_test version but was wondering why those methods are hanging off of the port object. We should probably look at the bigger picture of isolating test expectation related stuff into a separate class though. 540851 17 rniwa 2012-01-23 14:41:22 -0800 anyway, thanks for the review. 540853 18 rniwa 2012-01-23 14:42:15 -0800 Committed r105644: <http://trac.webkit.org/changeset/105644> 540859 19 dpranke 2012-01-23 14:46:30 -0800 (In reply to comment #16) > We should probably look at the bigger picture of isolating test expectation related stuff into a separate class though. There are (at least) two different operations going on ... the first is getting the content of things like test expectations files and skipped files, and that is (IMO) a port-specific operation, at the very least to figure out where the paths to the files are, and I try to go a step further so that you don't actually need a filesystem at all (you just ask for the a block of text data). The second is actually parsing those files to turn them into real objects, and that is already done in generic code (in models/test_expectations.py). Possibly it makes sense to add skipped file parsing logic to the latter. 123505 2012-01-22 17:59:44 -0800 2012-01-23 13:43:32 -0800 fixes the bug bug-76764-20120122175943.patch text/plain 6929 rniwa SW5kZXg6IFRvb2xzL0NoYW5nZUxvZwo9PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09Ci0tLSBUb29scy9DaGFuZ2VMb2cJKHJl dmlzaW9uIDEwNTU5MSkKKysrIFRvb2xzL0NoYW5nZUxvZwkod29ya2luZyBjb3B5KQpAQCAtMSwz ICsxLDI4IEBACisyMDEyLTAxLTIyICBSeW9zdWtlIE5pd2EgIDxybml3YUB3ZWJraXQub3JnPgor CisgICAgICAgIHJ1bi1wZXJmLXRlc3RzIGlnbm9yZSBTa2lwcGVkIGxpc3Qgb24gY2hyb21pdW0K KyAgICAgICAgaHR0cHM6Ly9idWdzLndlYmtpdC5vcmcvc2hvd19idWcuY2dpP2lkPTc2NzY0CisK KyAgICAgICAgUmV2aWV3ZWQgYnkgTk9CT0RZIChPT1BTISkuCisKKyAgICAgICAgTW92ZSBza2lw cGVkX3BlcmZfdGVzdHMgZnJvbSBXZWJLaXQgcG9ydCB0byBCYXNlIHBvcnQgc28gdGhhdCBDaHJv bWl1bSBwb3J0CisgICAgICAgIGNhbiBhbHNvIGZpbmQgc2tpcHBlZCBsaXN0LiBDaHJvbWl1bSBw b3J0IG9ubHkgdXNlcyB0ZXN0X2V4cGVjdGF0aW9ucy50eHQgZm9yCisgICAgICAgIGxheW91dCB0 ZXN0cyBidXQgcGVyZm9ybWFjbmUgdGVzdHMgZG9uJ3QgdXNlIHRlc3RfZXhwZWN0YXRpb25zLnR4 dCBzbyBDaHJvbWl1bSBwb3J0CisgICAgICAgIGFsc28gbmVlZHMgdG8gdXNlIFNraXBwZWQgbGlz dC4KKworICAgICAgICAqIFNjcmlwdHMvd2Via2l0cHkvbGF5b3V0X3Rlc3RzL3BvcnQvYmFzZS5w eToKKyAgICAgICAgKFBvcnQuX3Rlc3RzX2Zyb21fc2tpcHBlZF9maWxlX2NvbnRlbnRzKToKKyAg ICAgICAgKFBvcnQpOgorICAgICAgICAoUG9ydC5fZXhwZWN0YXRpb25zX2Zyb21fc2tpcHBlZF9m aWxlcyk6CisgICAgICAgIChQb3J0LnNraXBwZWRfcGVyZl90ZXN0cyk6CisgICAgICAgICogU2Ny aXB0cy93ZWJraXRweS9sYXlvdXRfdGVzdHMvcG9ydC9iYXNlX3VuaXR0ZXN0LnB5OgorICAgICAg ICAoUG9ydFRlc3QudGVzdF9za2lwcGVkX3BlcmZfdGVzdHMpOgorICAgICAgICAoUG9ydFRlc3Qu dGVzdF9za2lwcGVkX3BlcmZfdGVzdHMuYWRkX3RleHRfZmlsZSk6CisgICAgICAgICogU2NyaXB0 cy93ZWJraXRweS9sYXlvdXRfdGVzdHMvcG9ydC93ZWJraXQucHk6CisgICAgICAgIChXZWJLaXRQ b3J0Ll9za2lwcGVkX3Rlc3RzX2Zvcl91bnN1cHBvcnRlZF9mZWF0dXJlcyk6CisgICAgICAgIChX ZWJLaXRQb3J0Ll9za2lwcGVkX2ZpbGVfc2VhcmNoX3BhdGhzKToKKyAgICAgICAgKFdlYktpdFBv cnQuc2tpcHBlZF9sYXlvdXRfdGVzdHMpOgorCiAyMDEyLTAxLTIyICBTZXJnaW8gVmlsbGFyIFNl bmluICA8c3ZpbGxhckBpZ2FsaWEuY29tPgogCiAgICAgICAgIFtHVEtdIER1bXBSZW5kZXJUcmVl IGNvbnZlcnRzICJmaWxlOi8vLyIgdG8gYSBwYXRoIGRpZmZlcmVudGx5CkluZGV4OiBUb29scy9T Y3JpcHRzL3dlYmtpdHB5L2xheW91dF90ZXN0cy9wb3J0L2Jhc2UucHkKPT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PQotLS0g VG9vbHMvU2NyaXB0cy93ZWJraXRweS9sYXlvdXRfdGVzdHMvcG9ydC9iYXNlLnB5CShyZXZpc2lv biAxMDU1NTMpCisrKyBUb29scy9TY3JpcHRzL3dlYmtpdHB5L2xheW91dF90ZXN0cy9wb3J0L2Jh c2UucHkJKHdvcmtpbmcgY29weSkKQEAgLTU3NCw4ICs1NzQsMzEgQEAgY2xhc3MgUG9ydChvYmpl Y3QpOgogICAgIGRlZiBza2lwcGVkX2xheW91dF90ZXN0cyhzZWxmKToKICAgICAgICAgcmV0dXJu IFtdCiAKKyAgICBkZWYgX3Rlc3RzX2Zyb21fc2tpcHBlZF9maWxlX2NvbnRlbnRzKHNlbGYsIHNr aXBwZWRfZmlsZV9jb250ZW50cyk6CisgICAgICAgIHRlc3RzX3RvX3NraXAgPSBbXQorICAgICAg ICBmb3IgbGluZSBpbiBza2lwcGVkX2ZpbGVfY29udGVudHMuc3BsaXQoJ1xuJyk6CisgICAgICAg ICAgICBsaW5lID0gbGluZS5zdHJpcCgpCisgICAgICAgICAgICBsaW5lID0gbGluZS5yc3RyaXAo Jy8nKSAgIyBCZXN0IHRvIG5vcm1hbGl6ZSBkaXJlY3RvcnkgbmFtZXMgdG8gbm90IGluY2x1ZGUg dGhlIHRyYWlsaW5nIHNsYXNoLgorICAgICAgICAgICAgaWYgbGluZS5zdGFydHN3aXRoKCcjJykg b3Igbm90IGxlbihsaW5lKToKKyAgICAgICAgICAgICAgICBjb250aW51ZQorICAgICAgICAgICAg dGVzdHNfdG9fc2tpcC5hcHBlbmQobGluZSkKKyAgICAgICAgcmV0dXJuIHRlc3RzX3RvX3NraXAK KworICAgIGRlZiBfZXhwZWN0YXRpb25zX2Zyb21fc2tpcHBlZF9maWxlcyhzZWxmLCBza2lwcGVk X2ZpbGVfcGF0aHMpOgorICAgICAgICB0ZXN0c190b19za2lwID0gW10KKyAgICAgICAgZm9yIHNl YXJjaF9wYXRoIGluIHNraXBwZWRfZmlsZV9wYXRoczoKKyAgICAgICAgICAgIGZpbGVuYW1lID0g c2VsZi5fZmlsZXN5c3RlbS5qb2luKHNlbGYuX3dlYmtpdF9iYXNlbGluZV9wYXRoKHNlYXJjaF9w YXRoKSwgIlNraXBwZWQiKQorICAgICAgICAgICAgaWYgbm90IHNlbGYuX2ZpbGVzeXN0ZW0uZXhp c3RzKGZpbGVuYW1lKToKKyAgICAgICAgICAgICAgICBfbG9nLmRlYnVnKCJTa2lwcGVkIGRvZXMg bm90IGV4aXN0OiAlcyIgJSBmaWxlbmFtZSkKKyAgICAgICAgICAgICAgICBjb250aW51ZQorICAg ICAgICAgICAgX2xvZy5kZWJ1ZygiVXNpbmcgU2tpcHBlZCBmaWxlOiAlcyIgJSBmaWxlbmFtZSkK KyAgICAgICAgICAgIHNraXBwZWRfZmlsZV9jb250ZW50cyA9IHNlbGYuX2ZpbGVzeXN0ZW0ucmVh ZF90ZXh0X2ZpbGUoZmlsZW5hbWUpCisgICAgICAgICAgICB0ZXN0c190b19za2lwLmV4dGVuZChz ZWxmLl90ZXN0c19mcm9tX3NraXBwZWRfZmlsZV9jb250ZW50cyhza2lwcGVkX2ZpbGVfY29udGVu dHMpKQorICAgICAgICByZXR1cm4gdGVzdHNfdG9fc2tpcAorCisgICAgQG1lbW9pemVkCiAgICAg ZGVmIHNraXBwZWRfcGVyZl90ZXN0cyhzZWxmKToKLSAgICAgICAgcmV0dXJuIFtdCisgICAgICAg IHJldHVybiBzZWxmLl9leHBlY3RhdGlvbnNfZnJvbV9za2lwcGVkX2ZpbGVzKFtzZWxmLnBlcmZf dGVzdHNfZGlyKCldKQogCiAgICAgZGVmIHNraXBwZWRfdGVzdHMoc2VsZik6CiAgICAgICAgIHJl dHVybiBbXQpJbmRleDogVG9vbHMvU2NyaXB0cy93ZWJraXRweS9sYXlvdXRfdGVzdHMvcG9ydC9i YXNlX3VuaXR0ZXN0LnB5Cj09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT0KLS0tIFRvb2xzL1NjcmlwdHMvd2Via2l0cHkvbGF5 b3V0X3Rlc3RzL3BvcnQvYmFzZV91bml0dGVzdC5weQkocmV2aXNpb24gMTA1NTUzKQorKysgVG9v bHMvU2NyaXB0cy93ZWJraXRweS9sYXlvdXRfdGVzdHMvcG9ydC9iYXNlX3VuaXR0ZXN0LnB5CSh3 b3JraW5nIGNvcHkpCkBAIC0yMzAsNiArMjMwLDIyIEBAIGNsYXNzIFBvcnRUZXN0KHVuaXR0ZXN0 LlRlc3RDYXNlKToKICAgICAgICAgc2VsZi5hc3NlcnRUcnVlKCdjYW52YXMnIGluIGRpcnMpCiAg ICAgICAgIHNlbGYuYXNzZXJ0VHJ1ZSgnY3NzMi4xJyBpbiBkaXJzKQogCisgICAgZGVmIHRlc3Rf c2tpcHBlZF9wZXJmX3Rlc3RzKHNlbGYpOgorICAgICAgICBwb3J0ID0gc2VsZi5tYWtlX3BvcnQo KQorCisgICAgICAgIGRlZiBhZGRfdGV4dF9maWxlKGRpcm5hbWUsIGZpbGVuYW1lLCBjb250ZW50 PSdzb21lIGNvbnRlbnQnKToKKyAgICAgICAgICAgIGRpcm5hbWUgPSBwb3J0Lmhvc3QuZmlsZXN5 c3RlbS5qb2luKHBvcnQucGVyZl90ZXN0c19kaXIoKSwgZGlybmFtZSkKKyAgICAgICAgICAgIHBv cnQuaG9zdC5maWxlc3lzdGVtLm1heWJlX21ha2VfZGlyZWN0b3J5KGRpcm5hbWUpCisgICAgICAg ICAgICBwb3J0Lmhvc3QuZmlsZXN5c3RlbS53cml0ZV90ZXh0X2ZpbGUocG9ydC5ob3N0LmZpbGVz eXN0ZW0uam9pbihkaXJuYW1lLCBmaWxlbmFtZSksIGNvbnRlbnQpCisKKyAgICAgICAgYWRkX3Rl eHRfZmlsZSgnaW5zcGVjdG9yJywgJ3Rlc3QxLmh0bWwnKQorICAgICAgICBhZGRfdGV4dF9maWxl KCdpbnNwZWN0b3InLCAndW5zdXBwb3J0ZWRfdGVzdDEuaHRtbCcpCisgICAgICAgIGFkZF90ZXh0 X2ZpbGUoJ2luc3BlY3RvcicsICd0ZXN0Mi5odG1sJykKKyAgICAgICAgYWRkX3RleHRfZmlsZSgn aW5zcGVjdG9yL3Jlc291cmNlcycsICdyZXNvdXJjZV9maWxlLmh0bWwnKQorICAgICAgICBhZGRf dGV4dF9maWxlKCd1bnN1cHBvcnRlZCcsICd1bnN1cHBvcnRlZF90ZXN0Mi5odG1sJykKKyAgICAg ICAgYWRkX3RleHRfZmlsZSgnJywgJ1NraXBwZWQnLCAnXG4nLmpvaW4oWydMYXlvdXQnLCAnJywg J1N1blNwaWRlcicsICdTdXBwb3J0ZWQvc29tZS10ZXN0Lmh0bWwnXSkpCisgICAgICAgIHNlbGYu YXNzZXJ0RXF1YWwocG9ydC5za2lwcGVkX3BlcmZfdGVzdHMoKSwgWydMYXlvdXQnLCAnU3VuU3Bp ZGVyJywgJ1N1cHBvcnRlZC9zb21lLXRlc3QuaHRtbCddKQorCiAgICAgZGVmIHRlc3RfZ2V0X29w dGlvbl9fc2V0KHNlbGYpOgogICAgICAgICBvcHRpb25zLCBhcmdzID0gb3B0cGFyc2UuT3B0aW9u UGFyc2VyKCkucGFyc2VfYXJncyhbXSkKICAgICAgICAgb3B0aW9ucy5mb28gPSAnYmFyJwpJbmRl eDogVG9vbHMvU2NyaXB0cy93ZWJraXRweS9sYXlvdXRfdGVzdHMvcG9ydC93ZWJraXQucHkKPT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PQotLS0gVG9vbHMvU2NyaXB0cy93ZWJraXRweS9sYXlvdXRfdGVzdHMvcG9ydC93ZWJr aXQucHkJKHJldmlzaW9uIDEwNTU1MykKKysrIFRvb2xzL1NjcmlwdHMvd2Via2l0cHkvbGF5b3V0 X3Rlc3RzL3BvcnQvd2Via2l0LnB5CSh3b3JraW5nIGNvcHkpCkBAIC0zMjIsMTYgKzMyMiw2IEBA IGNsYXNzIFdlYktpdFBvcnQoUG9ydCk6CiAgICAgICAgICMgRmFpbGVkIHRvIGdldCBhbnkgcnVu dGltZSBvciBzeW1ib2wgaW5mb3JtYXRpb24sIGRvbid0IHNraXAgYW55IHRlc3RzLgogICAgICAg ICByZXR1cm4gW10KIAotICAgIGRlZiBfdGVzdHNfZnJvbV9za2lwcGVkX2ZpbGVfY29udGVudHMo c2VsZiwgc2tpcHBlZF9maWxlX2NvbnRlbnRzKToKLSAgICAgICAgdGVzdHNfdG9fc2tpcCA9IFtd Ci0gICAgICAgIGZvciBsaW5lIGluIHNraXBwZWRfZmlsZV9jb250ZW50cy5zcGxpdCgnXG4nKToK LSAgICAgICAgICAgIGxpbmUgPSBsaW5lLnN0cmlwKCkKLSAgICAgICAgICAgIGxpbmUgPSBsaW5l LnJzdHJpcCgnLycpICAjIEJlc3QgdG8gbm9ybWFsaXplIGRpcmVjdG9yeSBuYW1lcyB0byBub3Qg aW5jbHVkZSB0aGUgdHJhaWxpbmcgc2xhc2guCi0gICAgICAgICAgICBpZiBsaW5lLnN0YXJ0c3dp dGgoJyMnKSBvciBub3QgbGVuKGxpbmUpOgotICAgICAgICAgICAgICAgIGNvbnRpbnVlCi0gICAg ICAgICAgICB0ZXN0c190b19za2lwLmFwcGVuZChsaW5lKQotICAgICAgICByZXR1cm4gdGVzdHNf dG9fc2tpcAotCiAgICAgZGVmIF93azJfcG9ydF9uYW1lKHNlbGYpOgogICAgICAgICAjIEJ5IGN1 cnJlbnQgY29udmVudGlvbiwgdGhlIFdlYktpdDIgbmFtZSBpcyBhbHdheXMgbWFjLXdrMiwgd2lu LXdrMiwgbm90IG1hYy1sZW9wYXJkLXdrMiwgZXRjLgogICAgICAgICByZXR1cm4gIiVzLXdrMiIg JSBzZWxmLnBvcnRfbmFtZQpAQCAtMzUxLDE4ICszNDEsNiBAQCBjbGFzcyBXZWJLaXRQb3J0KFBv cnQpOgogICAgICAgICAgICAgc2VhcmNoX3BhdGhzLnVwZGF0ZShbc2VsZi5fd2syX3BvcnRfbmFt ZSgpLCAid2syIl0pCiAgICAgICAgIHJldHVybiBzZWFyY2hfcGF0aHMKIAotICAgIGRlZiBfZXhw ZWN0YXRpb25zX2Zyb21fc2tpcHBlZF9maWxlcyhzZWxmLCBza2lwcGVkX2ZpbGVfcGF0aHMpOgot ICAgICAgICB0ZXN0c190b19za2lwID0gW10KLSAgICAgICAgZm9yIHNlYXJjaF9wYXRoIGluIHNr aXBwZWRfZmlsZV9wYXRoczoKLSAgICAgICAgICAgIGZpbGVuYW1lID0gc2VsZi5fZmlsZXN5c3Rl bS5qb2luKHNlbGYuX3dlYmtpdF9iYXNlbGluZV9wYXRoKHNlYXJjaF9wYXRoKSwgIlNraXBwZWQi KQotICAgICAgICAgICAgaWYgbm90IHNlbGYuX2ZpbGVzeXN0ZW0uZXhpc3RzKGZpbGVuYW1lKToK LSAgICAgICAgICAgICAgICBfbG9nLmRlYnVnKCJTa2lwcGVkIGRvZXMgbm90IGV4aXN0OiAlcyIg JSBmaWxlbmFtZSkKLSAgICAgICAgICAgICAgICBjb250aW51ZQotICAgICAgICAgICAgX2xvZy5k ZWJ1ZygiVXNpbmcgU2tpcHBlZCBmaWxlOiAlcyIgJSBmaWxlbmFtZSkKLSAgICAgICAgICAgIHNr aXBwZWRfZmlsZV9jb250ZW50cyA9IHNlbGYuX2ZpbGVzeXN0ZW0ucmVhZF90ZXh0X2ZpbGUoZmls ZW5hbWUpCi0gICAgICAgICAgICB0ZXN0c190b19za2lwLmV4dGVuZChzZWxmLl90ZXN0c19mcm9t X3NraXBwZWRfZmlsZV9jb250ZW50cyhza2lwcGVkX2ZpbGVfY29udGVudHMpKQotICAgICAgICBy ZXR1cm4gdGVzdHNfdG9fc2tpcAotCiAgICAgZGVmIHRlc3RfZXhwZWN0YXRpb25zKHNlbGYpOgog ICAgICAgICAjIFRoaXMgYWxsb3dzIHBvcnRzIHRvIHVzZSBhIGNvbWJpbmF0aW9uIG9mIHRlc3Rf ZXhwZWN0YXRpb25zLnR4dCBmaWxlcyBhbmQgU2tpcHBlZCBsaXN0cy4KICAgICAgICAgZXhwZWN0 YXRpb25zID0gJycKQEAgLTM4MCwxMCArMzU4LDYgQEAgY2xhc3MgV2ViS2l0UG9ydChQb3J0KToK ICAgICAgICAgdGVzdHNfdG9fc2tpcC51cGRhdGUoc2VsZi5fc2tpcHBlZF90ZXN0c19mb3JfdW5z dXBwb3J0ZWRfZmVhdHVyZXMoKSkKICAgICAgICAgcmV0dXJuIHRlc3RzX3RvX3NraXAKIAotICAg IEBtZW1vaXplZAotICAgIGRlZiBza2lwcGVkX3BlcmZfdGVzdHMoc2VsZik6Ci0gICAgICAgIHJl dHVybiBzZWxmLl9leHBlY3RhdGlvbnNfZnJvbV9za2lwcGVkX2ZpbGVzKFtzZWxmLnBlcmZfdGVz dHNfZGlyKCldKQotCiAgICAgZGVmIHNraXBwZWRfdGVzdHMoc2VsZik6CiAgICAgICAgIHJldHVy biBzZWxmLnNraXBwZWRfbGF5b3V0X3Rlc3RzKCkKIAo=