RESOLVED FIXED 38296
Import Sputnik JavaScript test suite 
https://bugs.webkit.org/show_bug.cgi?id=38296
Summary Import Sputnik JavaScript test suite
Alexey Proskuryakov
Reported 2010-04-28 17:21:54 PDT
Created attachment 54639 [details] translation script Some notes: - This contains both conformance tests that the web page runs, and other tests from source checkout. - It would be better to use run-javascriptcore-tests for these tests, but also more difficult to make a nice test runner. Someone can do that later. - Sputnik checkout contains tests that need to be processed before use. I've processed them to something like script-tests, but monolithic for better loading speed. - Sputnik tests often have several subtests, but bail out after the first error. There seems to be work in progress to distinguish hard failures from simple ones, and some tests use FAIL and ERROR distinctly - but since Google's own runner doesn't seem to distinguish those, I also didn't. - Several Unicode tests are disabled, because they are too slow due to iterating lots of code points. To import the tests, one needs to get a source checkout as below, edit sputnik.py, and run the attached script. - self.test_root = path.join(root, 'tests', 'Conformance') + self.test_root = path.join(root, 'tests')
Attachments
translation script (3.82 KB, text/plain)
2010-04-28 17:21 PDT, Alexey Proskuryakov 		no flags
Details
partial patch (101.56 KB, patch)
2010-04-28 17:33 PDT, Alexey Proskuryakov 		darin: review+
DetailsFormatted DiffDiff
View All Add attachment proposed patch, testcase, etc.
Alexey Proskuryakov
Comment 1 2010-04-28 17:33:30 PDT
Created attachment 54642 [details] partial patch The complete patch is 23 megabytes, uploading a part.
Darin Adler
Comment 2 2010-04-28 17:36:03 PDT
Comment on attachment 54642 [details] partial patch rs=me Is there a way to reduce the total amount of boilerplate? Since this suite was made by the V8 team, I assume that the failures are JavaScriptCore-specific, so we need to expect success on the Chromium platform. How will we do that?
Alexey Proskuryakov
Comment 3 2010-04-28 17:45:15 PDT
> Is there a way to reduce the total amount of boilerplate? It was my intentional choice to have monolithic tests for faster loads (but I didn't measure how much that helps). But I don't feel very strongly about either approach - hopefully, these tests will be moved to run-javascriptcore-tests in foreseeable future. > we need to expect success on the Chromium platform. How will we do that? Someone from Chromium team usually updates expectations later, since build.webkit.org doesn't list Chromium results, as far as I can tell. I'm quite worried about timeouts on slower bots though - there are several tests that took noticeable time on my Mac Pro. V8 fails more of these tests than JSC does, according to <http://sputnik.googlelabs.com/compare>.
Alexey Proskuryakov
Comment 4 2010-04-29 13:47:06 PDT
Committed <http://trac.webkit.org/changeset/58534>.
WebKit Review Bot
Comment 5 2010-04-29 14:07:48 PDT
http://trac.webkit.org/changeset/58534 might have broken GTK Linux 64-bit Release and Qt Linux Release The following changes are on the blame list: http://trac.webkit.org/changeset/58533 http://trac.webkit.org/changeset/58534
Alexey Proskuryakov
Comment 6 2010-04-29 14:36:22 PDT
Added the whole directory to Skipped list, since there are mysterious (and different) failures on all platforms besides SnowLeopard.
Alexey Proskuryakov
Comment 7 2010-04-29 17:01:46 PDT
Paths in the test suite are way too long for svn on Windows. Subversion only supports up to 256 or 260 when used with relative paths, but up to 32K with absolute paths. I had to shorten the paths manually for checkout to work.
Alexey Proskuryakov
Comment 8 2010-04-29 17:11:36 PDT
Filed http://code.google.com/p/sputniktests/issues/detail?id=25
Note You need to log in before you can comment on or make changes to this bug.