Bug 188450

Summary: test262-runner -s --test-only should replace test results
Product: WebKit Reporter: Keith Miller <keith_miller>
Component: Tools / TestsAssignee: Keith Miller <keith_miller>
Status: RESOLVED FIXED    
Severity: Normal CC: commit-queue, keith_miller, leo, lforschler, mcatanzaro, msaboff, webkit-bug-importer
Priority: P2 Keywords: InRadar
Version: WebKit Nightly Build   
Hardware: Unspecified   
OS: Unspecified   
Attachments:
Description Flags
Patch none

Description Keith Miller 2018-08-09 14:08:20 PDT
Right now if you run the test262-runner and tell it to save results while only running a subset of the tests it will clear all the other results. It should probably regex replace the results for what actually changed.
Comment 1 Keith Miller 2018-08-10 14:58:32 PDT
Created attachment 346931 [details]
Patch
Comment 2 Michael Saboff 2018-08-21 10:00:45 PDT
Comment on attachment 346931 [details]
Patch

r=me
Comment 3 WebKit Commit Bot 2018-08-27 04:49:16 PDT
Comment on attachment 346931 [details]
Patch

Clearing flags on attachment: 346931

Committed r235359: <https://trac.webkit.org/changeset/235359>
Comment 4 WebKit Commit Bot 2018-08-27 04:49:17 PDT
All reviewed patches have been landed.  Closing bug.
Comment 5 Radar WebKit Bug Importer 2018-08-27 04:50:17 PDT
<rdar://problem/43750234>
Comment 6 Michael Catanzaro 2018-09-04 08:49:57 PDT
This introduced 10 webkitperl test failures for WPE/GTK:

Tools/Scripts/webkitperl/test262_unittest/test262-runner-tests.pl .................................. Experimental keys on scalar is now forbidden at /home/slave/webkitgtk/gtk-linux-64-release-tests/build/Tools/Scripts/test262/Runner.pm line 929.
Compilation failed in require at /home/slave/webkitgtk/gtk-linux-64-release-tests/build/Tools/Scripts/test262-runner line 45.
BEGIN failed--compilation aborted at /home/slave/webkitgtk/gtk-linux-64-release-tests/build/Tools/Scripts/test262-runner line 45.

Tools/Scripts/webkitperl/test262_unittest/test262-runner-tests.pl .................................. 1/13 
#   Failed test 'test262 test failed, ignore expectations (exit code: 1)'
#   at Tools/Scripts/webkitperl/test262_unittest/test262-runner-tests.pl line 136.
#          got: '255'
#     expected: '1'

#   Failed test 'test262 test failed, ignore expectations (new failures: 2)'
#   at Tools/Scripts/webkitperl/test262_unittest/test262-runner-tests.pl line 142.
#          got: '0'
#     expected: '2'
Experimental keys on scalar is now forbidden at /home/slave/webkitgtk/gtk-linux-64-release-tests/build/Tools/Scripts/test262/Runner.pm line 929.
Compilation failed in require at /home/slave/webkitgtk/gtk-linux-64-release-tests/build/Tools/Scripts/test262-runner line 45.
BEGIN failed--compilation aborted at /home/slave/webkitgtk/gtk-linux-64-release-tests/build/Tools/Scripts/test262-runner line 45.

#   Failed test 'test262 test passed, ignore expectations (exit code: 0)'
#   at Tools/Scripts/webkitperl/test262_unittest/test262-runner-tests.pl line 136.
#          got: '255'
#     expected: '0'
Experimental keys on scalar is now forbidden at /home/slave/webkitgtk/gtk-linux-64-release-tests/build/Tools/Scripts/test262/Runner.pm line 929.
Compilation failed in require at /home/slave/webkitgtk/gtk-linux-64-release-tests/build/Tools/Scripts/test262-runner line 45.
BEGIN failed--compilation aborted at /home/slave/webkitgtk/gtk-linux-64-release-tests/build/Tools/Scripts/test262-runner line 45.

#   Failed test 'test262 tests newly failed (exit code: 1)'
#   at Tools/Scripts/webkitperl/test262_unittest/test262-runner-tests.pl line 136.
#          got: '255'
#     expected: '1'

#   Failed test 'test262 tests newly failed (new failures: 2)'
#   at Tools/Scripts/webkitperl/test262_unittest/test262-runner-tests.pl line 142.
#          got: '0'
#     expected: '2'
Experimental keys on scalar is now forbidden at /home/slave/webkitgtk/gtk-linux-64-release-tests/build/Tools/Scripts/test262/Runner.pm line 929.
Compilation failed in require at /home/slave/webkitgtk/gtk-linux-64-release-tests/build/Tools/Scripts/test262-runner line 45.
BEGIN failed--compilation aborted at /home/slave/webkitgtk/gtk-linux-64-release-tests/build/Tools/Scripts/test262-runner line 45.

#   Failed test 'test262 tests newly passed (exit code: 0)'
#   at Tools/Scripts/webkitperl/test262_unittest/test262-runner-tests.pl line 136.
#          got: '255'
#     expected: '0'
Experimental keys on scalar is now forbidden at /home/slave/webkitgtk/gtk-linux-64-release-tests/build/Tools/Scripts/test262/Runner.pm line 929.
Compilation failed in require at /home/slave/webkitgtk/gtk-linux-64-release-tests/build/Tools/Scripts/test262-runner line 45.
BEGIN failed--compilation aborted at /home/slave/webkitgtk/gtk-linux-64-release-tests/build/Tools/Scripts/test262-runner line 45.

#   Failed test 'test262 tests fails, expected failure (exit code: 0)'
#   at Tools/Scripts/webkitperl/test262_unittest/test262-runner-tests.pl line 136.
#          got: '255'
#     expected: '0'
Experimental keys on scalar is now forbidden at /home/slave/webkitgtk/gtk-linux-64-release-tests/build/Tools/Scripts/test262/Runner.pm line 929.
Compilation failed in require at /home/slave/webkitgtk/gtk-linux-64-release-tests/build/Tools/Scripts/test262-runner line 45.
BEGIN failed--compilation aborted at /home/slave/webkitgtk/gtk-linux-64-release-tests/build/Tools/Scripts/test262-runner line 45.

#   Failed test 'test262 tests fails, with unexpected error string (exit code: 1)'
#   at Tools/Scripts/webkitperl/test262_unittest/test262-runner-tests.pl line 136.
#          got: '255'
#     expected: '1'

#   Failed test 'test262 tests fails, with unexpected error string (new failures: 2)'
#   at Tools/Scripts/webkitperl/test262_unittest/test262-runner-tests.pl line 142.
#          got: '0'
#     expected: '2'
Experimental keys on scalar is now forbidden at /home/slave/webkitgtk/gtk-linux-64-release-tests/build/Tools/Scripts/test262/Runner.pm line 929.
Compilation failed in require at /home/slave/webkitgtk/gtk-linux-64-release-tests/build/Tools/Scripts/test262-runner line 45.
BEGIN failed--compilation aborted at /home/slave/webkitgtk/gtk-linux-64-release-tests/build/Tools/Scripts/test262-runner line 45.

#   Failed test 'expectations yaml file format'
#   at Tools/Scripts/webkitperl/test262_unittest/test262-runner-tests.pl line 157.
# Looks like you failed 10 tests of 13.

                                                                                                            
Tools/Scripts/webkitperl/test262_unittest/test262-runner-tests.pl .................................. Dubious, test returned 10 (wstat 2560, 0xa00)
Failed 10/13 subtests
Comment 7 Keith Miller 2018-09-04 10:17:21 PDT
It seems that perl has deprecated calling keys on a reference. Maybe this will fix that?: https://bugs.webkit.org/show_bug.cgi?id=189261. idk how to perl...