56047 2011-03-09 13:28:32 -0800 new-run-webkit-tests causes several tests to fail on mac port 2011-04-01 19:45:15 -0700 1 1 1 Unclassified WebKit Tools / Tests 528+ (Nightly build) PC OS X 10.5 RESOLVED FIXED P2 Normal --- 56760 57662 34984 1 tonyg dpranke dpranke mihaip oldest_to_newest 364740 0 tonyg 2011-03-09 13:28:32 -0800 new-run-webkit-tests is waaay faster than run-webkit-tests. I'd love to be able to switch to it. However, there are several tests that fail or timeout on new-run-webkit-tests, but pass under run-webkit-tests. This is probably closely related to bug 35358, but that only refers to timeouts. Bug 35266 might also help the ones that timeout. Here's a list from r79638 on the mac port: fast/canvas/webgl/uninitialized-test.html Test timed out Text diff mismatch expected actual diff wdiff pretty diff fast/css/font-face-in-shadow-DOM.html Text diff mismatch expected actual diff wdiff pretty diff fast/forms/restore-selection-after-layout.html Text diff mismatch expected actual diff wdiff pretty diff fast/frames/lots-of-iframes.html Test timed out Text diff mismatch expected actual diff wdiff pretty diff fast/frames/lots-of-objects.html Test timed out Text diff mismatch expected actual diff wdiff pretty diff fast/overflow/lots-of-sibling-inline-boxes.html Test timed out Text diff mismatch expected actual diff wdiff pretty diff fast/parser/block-nesting-cap.html Test timed out Text diff mismatch expected actual diff wdiff pretty diff fast/wcss/wap-input-format.xhtml Text diff mismatch expected actual diff wdiff pretty diff fast/wcss/wap-input-required.xhtml Text diff mismatch expected actual diff wdiff pretty diff fast/workers/storage/use-same-database-in-page-and-workers.html DumpRenderTree crashed stack http/tests/navigation/image-load-in-unload-handler.html Text diff mismatch expected actual diff wdiff pretty diff http/tests/navigation/ping-cookie.html Text diff mismatch expected actual diff wdiff pretty diff http/tests/navigation/ping-same-origin.html Text diff mismatch expected actual diff wdiff pretty diff http/tests/security/isolatedWorld/all-window-properties.html Text diff mismatch expected actual diff wdiff pretty diff http/tests/security/isolatedWorld/all-window-prototypes.html Text diff mismatch expected actual diff wdiff pretty diff http/tests/xmlhttprequest/basic-auth-nopassword.html Text diff mismatch expected actual diff wdiff pretty diff java/lc3/JavaObject/JavaObjectToBoolean-001-n.html Text diff mismatch expected actual diff wdiff pretty diff java/lc3/JavaObject/JavaObjectToBoolean-002-n.html Text diff mismatch expected actual diff wdiff pretty diff java/lc3/JavaObject/JavaObjectToDouble-001.html Text diff mismatch expected actual diff wdiff pretty diff media/invalid-media-url-crash.html DumpRenderTree crashed stack platform/chromium-linux/fast/text/international/complex-joining-using-gpos.html Text diff mismatch expected actual diff wdiff pretty diff sputnik/Conformance/07_Lexical_Conventions/7.8_Literals/7.8.5_Regular_Expression_Literals/S7.8.5_A1.1_T2.html Test timed out Text diff mismatch expected actual diff wdiff pretty diff sputnik/Conformance/07_Lexical_Conventions/7.8_Literals/7.8.5_Regular_Expression_Literals/S7.8.5_A1.4_T2.html Test timed out Text diff mismatch expected actual diff wdiff pretty diff sputnik/Conformance/07_Lexical_Conventions/7.8_Literals/7.8.5_Regular_Expression_Literals/S7.8.5_A2.1_T2.html Test timed out Text diff mismatch expected actual diff wdiff pretty diff sputnik/Conformance/07_Lexical_Conventions/7.8_Literals/7.8.5_Regular_Expression_Literals/S7.8.5_A2.4_T2.html Test timed out Text diff mismatch expected actual diff wdiff pretty diff sputnik/Unicode/Unicode_218/S7.6_A5.3_T2.html Test timed out Text diff mismatch expected actual diff wdiff pretty diff transitions/cancel-transition.html Text diff mismatch expected actual diff wdiff pretty diff 364745 1 dpranke 2011-03-09 13:33:56 -0800 Yes, I've noticed this as well. 370805 2 dpranke 2011-03-21 11:57:59 -0700 A number of tests appear to be sensitive to the order they are run in, and for some reason NRWT is running the tests in descending alphabetical order inside a directory. I've filed bug 56760 to fix that. There are still some tests that are failing for other reasons, so I'm leaving this open for now. 378303 3 dpranke 2011-04-01 19:45:15 -0700 I believe all of the issues that were causing NRWT to not run cleanly (in a single thread, at least) on Mac SL have been resolved. I expect we will still need to shake out some flakiness as a result of contention between multiple processes, but I'll mark this as closed because I've fixed the main issues.