58625 2011-04-14 18:49:45 -0700 new-run-webkit-tests: runs tests in a different order than ORWT 2011-06-24 14:47:25 -0700 1 1 1 Unclassified WebKit Tools / Tests 528+ (Nightly build) PC OS X 10.5 RESOLVED WONTFIX P2 Normal --- 58293 58691 62753 34984 1 dpranke webkit-unassigned abarth aroben eric laszlo.gombos ojan tony oldest_to_newest 386283 0 dpranke 2011-04-14 18:49:45 -0700 It turns out that ORWT has a fancy sort function that is aware of numeric substrings in a test name, so that test-10.html will run after test-2.html. NRWT needs equal fanciness :) 386285 1 89717 dpranke 2011-04-14 18:54:02 -0700 Created attachment 89717 Patch 386287 2 89717 dpranke 2011-04-14 18:55:12 -0700 Comment on attachment 89717 Patch Not marking this for review just yet. While I think the patch is ready for review, I may want to leave this bug open as a tracking bug as there may be other issues affecting the test run order, and additional flakiness may ensue on either the mac or chromium bots as we change the order tests are run in. 386323 3 89717 eric 2011-04-14 20:21:15 -0700 Comment on attachment 89717 Patch Nice! 386894 4 dpranke 2011-04-15 13:55:55 -0700 moving the numeric compare patch into a sub-bug (bug 58691). 420972 5 dpranke 2011-06-14 20:06:09 -0700 Okay, I've worked out all of the issues causing us to run a different list of test in a different order than ORWT, and am filing them as dependent bugs here: 1) ORWT will run all of the tests in a directory before it descends into subdirectories. NRWT sorts things slightly differently to account for the chunking arguments (--run-chunk, --run-part, --batch-size, etc.) 2) ORWT uses "natural sorting" on filenames and path names 3) ORWT, for some reason, will sort directories such that "inspector-enabled" is run before "inspector". I can't really think of a good reason for this. 4) NRWT will sort the shards by the number of tests in the shard. ORWT, since it has no real concept of sharding, just runs them in sort order 5) NRWT doesn't support webarchives, and it also has some outdated logic for feature detection and skipping (e.g., for mathml, xhtmlmp). 6) NRWT can deal with MHT tests, ORWT can't 7) NRWT can run reftests, ORWT will just skip them 8) NRWT runs the http/ tests first, ORWT runs them last Issues (1), (2), and (3) will be addressed by the latest patch I'll post to bug 58691. Issue (5) will be addressed by the patch in bug 58293. I can't decide if it's really worth "fixing" (or at least changing) (4) and (8), but perhaps it makes sense to fix them in the nonsharded code path (what you get when you run in a single thread or in experimental fully parallel). Presumably we're not going to fix (6) and (7), but that seems wholly unrelated to this bug and I suggest we treat them separately. 424264 6 aroben 2011-06-20 22:22:37 -0700 (In reply to comment #5) > 8) NRWT runs the http/ tests first, ORWT runs them last This seems like an extremely user-visible issue when you have a small number of shards. I'd recommend fixing this one just to give people fewer things to complain about (especially the people who are mostly interested in seeing if "early" tests fail). 424678 7 dpranke 2011-06-21 12:45:27 -0700 (In reply to comment #6) > (In reply to comment #5) > > 8) NRWT runs the http/ tests first, ORWT runs them last > > This seems like an extremely user-visible issue when you have a small number of shards. I'd recommend fixing this one just to give people fewer things to complain about (especially the people who are mostly interested in seeing if "early" tests fail). I think this is much less of an issue when you are running multiple shards in parallel, in which I think you will probably still want to run the http tests as soon as possible, since they take so long. 424828 8 ojan 2011-06-21 16:03:11 -0700 I think it makes sense to special-case the single-process case to run the http tests last. If you have >2 shards, not doing them first causes the tests to take longer to run. 424830 9 dpranke 2011-06-21 16:05:49 -0700 (In reply to comment #8) > I think it makes sense to special-case the single-process case to run the http tests last. If you have >2 shards, not doing them first causes the tests to take longer to run. We could do that, but I don't expect people to run the single-process case very often, so I'm not sure if it's worth it. That said, it's probably a one-line change ... 427159 10 dpranke 2011-06-24 14:47:25 -0700 (In reply to comment #9) > (In reply to comment #8) > > I think it makes sense to special-case the single-process case to run the http tests last. If you have >2 shards, not doing them first causes the tests to take longer to run. > > We could do that, but I don't expect people to run the single-process case very often, so I'm not sure if it's worth it. That said, it's probably a one-line change ... Okay, given the changes I've been making so that we can run more than one http test concurrently, this isn't really a one-line change any more, if you want to run the non-http tests without holding the http lock. See the comments I've made in the patch on bug 63112, and I'm unconvinced that it's worth it. For now, I'm closing this as WONTFIX (and clearing the ownership). If anyone really thinks this needs to be fixed, please feel free to reopen. 89717 2011-04-14 18:54:02 -0700 2011-04-15 14:07:15 -0700 Patch bug-58625-20110414185401.patch text/plain 4578 dpranke U3VidmVyc2lvbiBSZXZpc2lvbjogODM4OTUKZGlmZiAtLWdpdCBhL1Rvb2xzL0NoYW5nZUxvZyBi L1Rvb2xzL0NoYW5nZUxvZwppbmRleCA1YjliOWI4ZjNkNmIzYWZmYTkxZWFjNGIyZGY2MmMyYWE0 YWQwNGUxLi5lMDU4MzMxOTMwMzQ3NTk3ZTlhNjIyMjQ3ZTU0Zjk0MzE1MzM2ZGI4IDEwMDY0NAot LS0gYS9Ub29scy9DaGFuZ2VMb2cKKysrIGIvVG9vbHMvQ2hhbmdlTG9nCkBAIC0xLDMgKzEsMTQg QEAKKzIwMTEtMDQtMTQgIERpcmsgUHJhbmtlICA8ZHByYW5rZUBjaHJvbWl1bS5vcmc+CisKKyAg ICAgICAgUmV2aWV3ZWQgYnkgTk9CT0RZIChPT1BTISkuCisKKyAgICAgICAgbmV3LXJ1bi13ZWJr aXQtdGVzdHM6IHNvcnQgdGVzdHMgd2l0aCBlbWJlZGRlZCBudW1lcmljCisgICAgICAgIHN1YnN0 cmluZ3MgaW50ZWxsaWdlbnRseQorICAgICAgICBodHRwczovL2J1Z3Mud2Via2l0Lm9yZy9zaG93 X2J1Zy5jZ2k/aWQ9NTg2MjUKKworICAgICAgICAqIFNjcmlwdHMvd2Via2l0cHkvbGF5b3V0X3Rl c3RzL2xheW91dF9wYWNrYWdlL3Rlc3RfcnVubmVyLnB5OgorICAgICAgICAqIFNjcmlwdHMvd2Vi a2l0cHkvbGF5b3V0X3Rlc3RzL2xheW91dF9wYWNrYWdlL3Rlc3RfcnVubmVyX3VuaXR0ZXN0LnB5 OgorCiAyMDExLTA0LTE0ICBFcmljIFNlaWRlbCAgPGVyaWNAd2Via2l0Lm9yZz4KIAogICAgICAg ICBSZXZpZXdlZCBieSBEaW1pdHJpIEdsYXprb3YuCmRpZmYgLS1naXQgYS9Ub29scy9TY3JpcHRz L3dlYmtpdHB5L2xheW91dF90ZXN0cy9sYXlvdXRfcGFja2FnZS90ZXN0X3J1bm5lci5weSBiL1Rv b2xzL1NjcmlwdHMvd2Via2l0cHkvbGF5b3V0X3Rlc3RzL2xheW91dF9wYWNrYWdlL3Rlc3RfcnVu bmVyLnB5CmluZGV4IGQ4MWI5Yzg4N2E0Mjg5NDA0Nzk1YTFjNmZjNmMwNzA0NDI1MjE0MTEuLjVi MDEwZjI2NGE4NzdiOTk2OGYxMGJhMjBlODk1MDI3Y2QxODBhYmYgMTAwNjQ0Ci0tLSBhL1Rvb2xz L1NjcmlwdHMvd2Via2l0cHkvbGF5b3V0X3Rlc3RzL2xheW91dF9wYWNrYWdlL3Rlc3RfcnVubmVy LnB5CisrKyBiL1Rvb2xzL1NjcmlwdHMvd2Via2l0cHkvbGF5b3V0X3Rlc3RzL2xheW91dF9wYWNr YWdlL3Rlc3RfcnVubmVyLnB5CkBAIC00Myw2ICs0Myw3IEBAIGltcG9ydCBsb2dnaW5nCiBpbXBv cnQgbWF0aAogaW1wb3J0IFF1ZXVlCiBpbXBvcnQgcmFuZG9tCitpbXBvcnQgcmUKIGltcG9ydCBz eXMKIGltcG9ydCB0aW1lCiAKQEAgLTI5NSw3ICsyOTYsNyBAQCBjbGFzcyBUZXN0UnVubmVyOgog ICAgICAgICBpZiBzZWxmLl9vcHRpb25zLnJhbmRvbWl6ZV9vcmRlcjoKICAgICAgICAgICAgIHJh bmRvbS5zaHVmZmxlKHNlbGYuX3Rlc3RfZmlsZXNfbGlzdCkKICAgICAgICAgZWxzZToKLSAgICAg ICAgICAgIHNlbGYuX3Rlc3RfZmlsZXNfbGlzdC5zb3J0KCkKKyAgICAgICAgICAgIHNlbGYuX3Rl c3RfZmlsZXNfbGlzdC5zb3J0KGNtcD1udW1lcmljX2NvbXBhcmUpCiAKICAgICAgICAgIyBJZiB0 aGUgdXNlciBzcGVjaWZpZXMgdGhleSBqdXN0IHdhbnQgdG8gcnVuIGEgc3Vic2V0IG9mIHRoZSB0 ZXN0cywKICAgICAgICAgIyBqdXN0IGdyYWIgYSBzdWJzZXQgb2YgdGhlIG5vbi1za2lwcGVkIHRl c3RzLgpAQCAtMTE1MCw3ICsxMTUxLDcgQEAgY2xhc3MgVGVzdFJ1bm5lcjoKICAgICA8aDI+JSh0 aXRsZSlzICglKHRpbWUpcyk8L2gyPgogICAgICAgICAiIiIgJSB7J3RpdGxlJzogdGl0bGUsICd0 aW1lJzogb3ZlcnJpZGVfdGltZSBvciB0aW1lLmFzY3RpbWUoKX0KIAotICAgICAgICBmb3IgdGVz dF9maWxlIGluIHNvcnRlZCh0ZXN0X2ZpbGVzKToKKyAgICAgICAgZm9yIHRlc3RfZmlsZSBpbiBz b3J0ZWQodGVzdF9maWxlcywgY21wPW51bWVyaWNfY29tcGFyZSk6CiAgICAgICAgICAgICB0ZXN0 X25hbWUgPSBzZWxmLl9wb3J0LnJlbGF0aXZlX3Rlc3RfZmlsZW5hbWUodGVzdF9maWxlKQogICAg ICAgICAgICAgdGVzdF91cmwgPSBzZWxmLl9wb3J0LmZpbGVuYW1lX3RvX3VyaSh0ZXN0X2ZpbGUp CiAgICAgICAgICAgICBwYWdlICs9IHUiPHA+PGEgaHJlZj0nJXMnPiVzPC9hPjxiciAvPlxuIiAl ICh0ZXN0X3VybCwgdGVzdF9uYW1lKQpAQCAtMTIyMCwzICsxMjIxLDMwIEBAIGRlZiByZWFkX3Rl c3RfZmlsZXMoZnMsIGZpbGVzKToKICAgICAgICAgICAgICAgICBfbG9nLmNyaXRpY2FsKCctLXRl c3QtbGlzdCBmaWxlICIlcyIgbm90IGZvdW5kJyAlIGZpbGUpCiAgICAgICAgICAgICByYWlzZQog ICAgIHJldHVybiB0ZXN0cworCisKK2RlZiBudW1lcmljX2NvbXBhcmUoeCwgeSk6CisgICAgIiIi Q29tcGFyZXMgdHdvIHN0cmluZ3Mgd2l0aCBlbWJlZGRlZCBudW1lcmljIHN1YnN0cmluZ3MgcHJv cGVybHkuIiIiCisgICAgeF9wYXJ0cyA9IHJlLnNwbGl0KHInKFxkKyknLCB4KQorICAgIHlfcGFy dHMgPSByZS5zcGxpdChyJyhcZCspJywgeSkKKworICAgICMgQ29tcGFyZSBvbmUgY2h1bmsgYXQg YSB0aW1lLgorICAgICMgRWFjaCBjaHVuayBpcyBlaXRoZXIgYWxsIG51bWVyaWMgZGlnaXRzLCBv ciBhbGwgbm90IG51bWVyaWMgZGlnaXRzLgorICAgIHdoaWxlIHhfcGFydHMgYW5kIHlfcGFydHM6 CisgICAgICAgIHhfcGFydCA9IHhfcGFydHMucG9wKDApCisgICAgICAgIHlfcGFydCA9IHlfcGFy dHMucG9wKDApCisgICAgICAgIGlmIHJlLm1hdGNoKCdeXGQnLCB4X3BhcnQpIGFuZCByZS5tYXRj aCgnXlxkJywgeV9wYXJ0KToKKyAgICAgICAgICAgIGlmIGludCh4X3BhcnQpIDwgaW50KHlfcGFy dCk6CisgICAgICAgICAgICAgICAgcmV0dXJuIC0xCisgICAgICAgICAgICBlbGlmIGludCh4X3Bh cnQpID4gaW50KHlfcGFydCk6CisgICAgICAgICAgICAgICAgcmV0dXJuIDEKKyAgICAgICAgZWxz ZToKKyAgICAgICAgICAgIGlmIHhfcGFydCA8IHlfcGFydDoKKyAgICAgICAgICAgICAgICByZXR1 cm4gLTEKKyAgICAgICAgICAgIGVsaWYgeF9wYXJ0ID4geV9wYXJ0OgorICAgICAgICAgICAgICAg IHJldHVybiAxCisgICAgaWYgeF9wYXJ0czoKKyAgICAgICAgcmV0dXJuIDEKKyAgICBlbGlmIHlf cGFydHM6CisgICAgICAgIHJldHVybiAtMQorICAgIHJldHVybiAwCmRpZmYgLS1naXQgYS9Ub29s cy9TY3JpcHRzL3dlYmtpdHB5L2xheW91dF90ZXN0cy9sYXlvdXRfcGFja2FnZS90ZXN0X3J1bm5l cl91bml0dGVzdC5weSBiL1Rvb2xzL1NjcmlwdHMvd2Via2l0cHkvbGF5b3V0X3Rlc3RzL2xheW91 dF9wYWNrYWdlL3Rlc3RfcnVubmVyX3VuaXR0ZXN0LnB5CmluZGV4IDk3Zjg2MzBmYTkwNzE0OTlk MmFhNTY0NzhiMTYxYjU2ODliYzY1NjkuLjhiZmE0MjIyMGQ2NTRjYTJlYjIyMWJhMDg4NGFhNTc4 ZTc5NWY0YzcgMTAwNjQ0Ci0tLSBhL1Rvb2xzL1NjcmlwdHMvd2Via2l0cHkvbGF5b3V0X3Rlc3Rz L2xheW91dF9wYWNrYWdlL3Rlc3RfcnVubmVyX3VuaXR0ZXN0LnB5CisrKyBiL1Rvb2xzL1Njcmlw dHMvd2Via2l0cHkvbGF5b3V0X3Rlc3RzL2xheW91dF9wYWNrYWdlL3Rlc3RfcnVubmVyX3VuaXR0 ZXN0LnB5CkBAIC0xMDIsNSArMTAyLDI3IEBAIGNsYXNzIFRlc3RSdW5uZXJUZXN0KHVuaXR0ZXN0 LlRlc3RDYXNlKToKICAgICAgICAgc2VsZi5hc3NlcnRFcXVhbChleHBlY3RlZF90ZXN0c190b19o dHRwX2xvY2ssIHNldChtdWx0aV90aHJlYWRfcmVzdWx0c1swXVsxXSkpCiAKIAorY2xhc3MgTnVt ZXJpY0NtcFRlc3QodW5pdHRlc3QuVGVzdENhc2UpOgorICAgIGRlZiBhc3NlcnRfY21wKHNlbGYs IHgsIHksIHJlc3VsdCk6CisgICAgICAgIHNlbGYuYXNzZXJ0RXF1YWxzKHRlc3RfcnVubmVyLm51 bWVyaWNfY29tcGFyZSh4LCB5KSwgcmVzdWx0KQorCisgICAgZGVmIHRlc3RfbnVtZXJpY19jbXAo c2VsZik6CisgICAgICAgIHNlbGYuYXNzZXJ0X2NtcCgnYScsICdhJywgMCkKKyAgICAgICAgc2Vs Zi5hc3NlcnRfY21wKCdhYicsICdhJywgMSkKKyAgICAgICAgc2VsZi5hc3NlcnRfY21wKCdhJywg J2FiJywgLTEpCisgICAgICAgIHNlbGYuYXNzZXJ0X2NtcCgnJywgJycsIDApCisgICAgICAgIHNl bGYuYXNzZXJ0X2NtcCgnJywgJ2FiJywgLTEpCisgICAgICAgIHNlbGYuYXNzZXJ0X2NtcCgnMScs ICcyJywgLTEpCisgICAgICAgIHNlbGYuYXNzZXJ0X2NtcCgnMicsICcxJywgMSkKKyAgICAgICAg c2VsZi5hc3NlcnRfY21wKCcxJywgJzEwJywgLTEpCisgICAgICAgIHNlbGYuYXNzZXJ0X2NtcCgn MicsICcxMCcsIC0xKQorICAgICAgICBzZWxmLmFzc2VydF9jbXAoJ2Zvb18xLmh0bWwnLCAnZm9v XzIuaHRtbCcsIC0xKQorICAgICAgICBzZWxmLmFzc2VydF9jbXAoJ2Zvb18xLjEuaHRtbCcsICdm b29fMi5odG1sJywgLTEpCisgICAgICAgIHNlbGYuYXNzZXJ0X2NtcCgnZm9vXzEuaHRtbCcsICdm b29fMTAuaHRtbCcsIC0xKQorICAgICAgICBzZWxmLmFzc2VydF9jbXAoJ2Zvb18yLmh0bWwnLCAn Zm9vXzEwLmh0bWwnLCAtMSkKKyAgICAgICAgc2VsZi5hc3NlcnRfY21wKCdmb29fMjMuaHRtbCcs ICdmb29fMTAuaHRtbCcsIDEpCisgICAgICAgIHNlbGYuYXNzZXJ0X2NtcCgnZm9vXzIzLmh0bWwn LCAnZm9vXzEwMC5odG1sJywgLTEpCisKKwogaWYgX19uYW1lX18gPT0gJ19fbWFpbl9fJzoKICAg ICB1bml0dGVzdC5tYWluKCkK