Bug 188450 - test262-runner -s --test-only should replace test results
Summary: test262-runner -s --test-only should replace test results
Status: RESOLVED FIXED
Alias: None
Product: WebKit
Classification: Unclassified
Component: Tools / Tests (show other bugs)
Version: WebKit Nightly Build
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: Keith Miller
URL:
Keywords: InRadar
Depends on:
Blocks:
 
Reported: 2018-08-09 14:08 PDT by Keith Miller
Modified: 2018-09-04 10:17 PDT (History)
7 users (show)

See Also:


Attachments
Patch (2.72 KB, patch)
2018-08-10 14:58 PDT, Keith Miller
no flags Details | Formatted Diff | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Keith Miller 2018-08-09 14:08:20 PDT
Right now if you run the test262-runner and tell it to save results while only running a subset of the tests it will clear all the other results. It should probably regex replace the results for what actually changed.
Comment 1 Keith Miller 2018-08-10 14:58:32 PDT
Created attachment 346931 [details]
Patch
Comment 2 Michael Saboff 2018-08-21 10:00:45 PDT
Comment on attachment 346931 [details]
Patch

r=me
Comment 3 WebKit Commit Bot 2018-08-27 04:49:16 PDT
Comment on attachment 346931 [details]
Patch

Clearing flags on attachment: 346931

Committed r235359: <https://trac.webkit.org/changeset/235359>
Comment 4 WebKit Commit Bot 2018-08-27 04:49:17 PDT
All reviewed patches have been landed.  Closing bug.
Comment 5 Radar WebKit Bug Importer 2018-08-27 04:50:17 PDT
<rdar://problem/43750234>
Comment 6 Michael Catanzaro 2018-09-04 08:49:57 PDT
This introduced 10 webkitperl test failures for WPE/GTK:

Tools/Scripts/webkitperl/test262_unittest/test262-runner-tests.pl .................................. Experimental keys on scalar is now forbidden at /home/slave/webkitgtk/gtk-linux-64-release-tests/build/Tools/Scripts/test262/Runner.pm line 929.
Compilation failed in require at /home/slave/webkitgtk/gtk-linux-64-release-tests/build/Tools/Scripts/test262-runner line 45.
BEGIN failed--compilation aborted at /home/slave/webkitgtk/gtk-linux-64-release-tests/build/Tools/Scripts/test262-runner line 45.

Tools/Scripts/webkitperl/test262_unittest/test262-runner-tests.pl .................................. 1/13 
#   Failed test 'test262 test failed, ignore expectations (exit code: 1)'
#   at Tools/Scripts/webkitperl/test262_unittest/test262-runner-tests.pl line 136.
#          got: '255'
#     expected: '1'

#   Failed test 'test262 test failed, ignore expectations (new failures: 2)'
#   at Tools/Scripts/webkitperl/test262_unittest/test262-runner-tests.pl line 142.
#          got: '0'
#     expected: '2'
Experimental keys on scalar is now forbidden at /home/slave/webkitgtk/gtk-linux-64-release-tests/build/Tools/Scripts/test262/Runner.pm line 929.
Compilation failed in require at /home/slave/webkitgtk/gtk-linux-64-release-tests/build/Tools/Scripts/test262-runner line 45.
BEGIN failed--compilation aborted at /home/slave/webkitgtk/gtk-linux-64-release-tests/build/Tools/Scripts/test262-runner line 45.

#   Failed test 'test262 test passed, ignore expectations (exit code: 0)'
#   at Tools/Scripts/webkitperl/test262_unittest/test262-runner-tests.pl line 136.
#          got: '255'
#     expected: '0'
Experimental keys on scalar is now forbidden at /home/slave/webkitgtk/gtk-linux-64-release-tests/build/Tools/Scripts/test262/Runner.pm line 929.
Compilation failed in require at /home/slave/webkitgtk/gtk-linux-64-release-tests/build/Tools/Scripts/test262-runner line 45.
BEGIN failed--compilation aborted at /home/slave/webkitgtk/gtk-linux-64-release-tests/build/Tools/Scripts/test262-runner line 45.

#   Failed test 'test262 tests newly failed (exit code: 1)'
#   at Tools/Scripts/webkitperl/test262_unittest/test262-runner-tests.pl line 136.
#          got: '255'
#     expected: '1'

#   Failed test 'test262 tests newly failed (new failures: 2)'
#   at Tools/Scripts/webkitperl/test262_unittest/test262-runner-tests.pl line 142.
#          got: '0'
#     expected: '2'
Experimental keys on scalar is now forbidden at /home/slave/webkitgtk/gtk-linux-64-release-tests/build/Tools/Scripts/test262/Runner.pm line 929.
Compilation failed in require at /home/slave/webkitgtk/gtk-linux-64-release-tests/build/Tools/Scripts/test262-runner line 45.
BEGIN failed--compilation aborted at /home/slave/webkitgtk/gtk-linux-64-release-tests/build/Tools/Scripts/test262-runner line 45.

#   Failed test 'test262 tests newly passed (exit code: 0)'
#   at Tools/Scripts/webkitperl/test262_unittest/test262-runner-tests.pl line 136.
#          got: '255'
#     expected: '0'
Experimental keys on scalar is now forbidden at /home/slave/webkitgtk/gtk-linux-64-release-tests/build/Tools/Scripts/test262/Runner.pm line 929.
Compilation failed in require at /home/slave/webkitgtk/gtk-linux-64-release-tests/build/Tools/Scripts/test262-runner line 45.
BEGIN failed--compilation aborted at /home/slave/webkitgtk/gtk-linux-64-release-tests/build/Tools/Scripts/test262-runner line 45.

#   Failed test 'test262 tests fails, expected failure (exit code: 0)'
#   at Tools/Scripts/webkitperl/test262_unittest/test262-runner-tests.pl line 136.
#          got: '255'
#     expected: '0'
Experimental keys on scalar is now forbidden at /home/slave/webkitgtk/gtk-linux-64-release-tests/build/Tools/Scripts/test262/Runner.pm line 929.
Compilation failed in require at /home/slave/webkitgtk/gtk-linux-64-release-tests/build/Tools/Scripts/test262-runner line 45.
BEGIN failed--compilation aborted at /home/slave/webkitgtk/gtk-linux-64-release-tests/build/Tools/Scripts/test262-runner line 45.

#   Failed test 'test262 tests fails, with unexpected error string (exit code: 1)'
#   at Tools/Scripts/webkitperl/test262_unittest/test262-runner-tests.pl line 136.
#          got: '255'
#     expected: '1'

#   Failed test 'test262 tests fails, with unexpected error string (new failures: 2)'
#   at Tools/Scripts/webkitperl/test262_unittest/test262-runner-tests.pl line 142.
#          got: '0'
#     expected: '2'
Experimental keys on scalar is now forbidden at /home/slave/webkitgtk/gtk-linux-64-release-tests/build/Tools/Scripts/test262/Runner.pm line 929.
Compilation failed in require at /home/slave/webkitgtk/gtk-linux-64-release-tests/build/Tools/Scripts/test262-runner line 45.
BEGIN failed--compilation aborted at /home/slave/webkitgtk/gtk-linux-64-release-tests/build/Tools/Scripts/test262-runner line 45.

#   Failed test 'expectations yaml file format'
#   at Tools/Scripts/webkitperl/test262_unittest/test262-runner-tests.pl line 157.
# Looks like you failed 10 tests of 13.

                                                                                                            
Tools/Scripts/webkitperl/test262_unittest/test262-runner-tests.pl .................................. Dubious, test returned 10 (wstat 2560, 0xa00)
Failed 10/13 subtests
Comment 7 Keith Miller 2018-09-04 10:17:21 PDT
It seems that perl has deprecated calling keys on a reference. Maybe this will fix that?: https://bugs.webkit.org/show_bug.cgi?id=189261. idk how to perl...