When a patch fails to build jsc, EWS should unapply the patch and build ToT in order to confirm that it is not a pre-existing issue (Similar to https://bugs.webkit.org/show_bug.cgi?id=187358). Also, when a patch fails jsc tests, EWS should re-run the tests (since these tests are flaky), and if the re-run still fails, it should un-apply the patch and re-run tests on clean tree (similar to current EWS).
Created attachment 344755 [details] Proposed patch We should also analyze the test results from all the re-runs, that would be in a separate step.
Tested in: http://ews-build.webkit-uat.org/#/builders/17/builds/23
Attachment 344755 [details] did not pass style-queue: ERROR: Tools/BuildSlaveSupport/ews-build/steps.py:213: [RunJavaScriptCoreTests.evaluateCommand] Use of super on an old style class [pylint/E1002] [5] ERROR: Tools/BuildSlaveSupport/ews-build/steps.py:229: [ReRunJavaScriptCoreTests.evaluateCommand] Use of super on an old style class [pylint/E1002] [5] ERROR: Tools/BuildSlaveSupport/ews-build/steps_unittest.py:548: [TestCompileJSCOnlyToT.test_success] Passing unexpected keyword argument 'state_string' in function call [pylint/E1123] [5] ERROR: Tools/BuildSlaveSupport/ews-build/steps_unittest.py:548: [TestCompileJSCOnlyToT.test_success] No value passed for parameter 'status_text' in function call [pylint/E1120] [5] ERROR: Tools/BuildSlaveSupport/ews-build/steps_unittest.py:563: [TestCompileJSCOnlyToT.test_failure] Passing unexpected keyword argument 'state_string' in function call [pylint/E1123] [5] ERROR: Tools/BuildSlaveSupport/ews-build/steps_unittest.py:563: [TestCompileJSCOnlyToT.test_failure] No value passed for parameter 'status_text' in function call [pylint/E1120] [5] ERROR: Tools/BuildSlaveSupport/ews-build/steps_unittest.py:571: [TestCompileJSCOnlyToT.test_skip] Passing unexpected keyword argument 'state_string' in function call [pylint/E1123] [5] ERROR: Tools/BuildSlaveSupport/ews-build/steps_unittest.py:571: [TestCompileJSCOnlyToT.test_skip] No value passed for parameter 'status_text' in function call [pylint/E1120] [5] ERROR: Tools/BuildSlaveSupport/ews-build/steps_unittest.py:635: [TestReRunJavaScriptCoreTests.test_success] Passing unexpected keyword argument 'state_string' in function call [pylint/E1123] [5] ERROR: Tools/BuildSlaveSupport/ews-build/steps_unittest.py:635: [TestReRunJavaScriptCoreTests.test_success] No value passed for parameter 'status_text' in function call [pylint/E1120] [5] ERROR: Tools/BuildSlaveSupport/ews-build/steps_unittest.py:651: [TestReRunJavaScriptCoreTests.test_failure] Passing unexpected keyword argument 'state_string' in function call [pylint/E1123] [5] ERROR: Tools/BuildSlaveSupport/ews-build/steps_unittest.py:651: [TestReRunJavaScriptCoreTests.test_failure] No value passed for parameter 'status_text' in function call [pylint/E1120] [5] ERROR: Tools/BuildSlaveSupport/ews-build/steps_unittest.py:659: [TestReRunJavaScriptCoreTests.test_skip] Passing unexpected keyword argument 'state_string' in function call [pylint/E1123] [5] ERROR: Tools/BuildSlaveSupport/ews-build/steps_unittest.py:659: [TestReRunJavaScriptCoreTests.test_skip] No value passed for parameter 'status_text' in function call [pylint/E1120] [5] ERROR: Tools/BuildSlaveSupport/ews-build/steps_unittest.py:684: [TestRunJavaScriptCoreTestsToT.test_success] Passing unexpected keyword argument 'state_string' in function call [pylint/E1123] [5] ERROR: Tools/BuildSlaveSupport/ews-build/steps_unittest.py:684: [TestRunJavaScriptCoreTestsToT.test_success] No value passed for parameter 'status_text' in function call [pylint/E1120] [5] ERROR: Tools/BuildSlaveSupport/ews-build/steps_unittest.py:700: [TestRunJavaScriptCoreTestsToT.test_failure] Passing unexpected keyword argument 'state_string' in function call [pylint/E1123] [5] ERROR: Tools/BuildSlaveSupport/ews-build/steps_unittest.py:700: [TestRunJavaScriptCoreTestsToT.test_failure] No value passed for parameter 'status_text' in function call [pylint/E1120] [5] ERROR: Tools/BuildSlaveSupport/ews-build/steps_unittest.py:708: [TestRunJavaScriptCoreTestsToT.test_skip] Passing unexpected keyword argument 'state_string' in function call [pylint/E1123] [5] ERROR: Tools/BuildSlaveSupport/ews-build/steps_unittest.py:708: [TestRunJavaScriptCoreTestsToT.test_skip] No value passed for parameter 'status_text' in function call [pylint/E1120] [5] Total errors found: 20 in 4 files If any of these errors are false positives, please file a bug against check-webkit-style.
Style checker warnings are false positive.
> We should also analyze the test results from all the re-runs, that would be in a separate step. I was exploring if Build Factory can access step results and do the analysis, but it seems like they can not be. See: https://lists.buildbot.net/pipermail/devel/2018-July/012543.html The other option suggested here is to have one step dynamically add next step, but I think in this case that would result in confusing and unreadable code. Adding a separate step to do the analysis of all the re-runs seems like the best option.
Comment on attachment 344755 [details] Proposed patch Clearing flags on attachment: 344755 Committed r233757: <https://trac.webkit.org/changeset/233757>
All reviewed patches have been landed. Closing bug.
<rdar://problem/42102573>