WebKit Bugzilla
New
Browse
Search+
Log In
×
Sign in with GitHub
or
Remember my login
Create Account
·
Forgot Password
Forgotten password account recovery
[patch]
Patch
bug-162458-20170202162537.patch (text/plain), 77.84 KB, created by
Srinivasan Vijayaraghavan
on 2017-02-02 16:27:51 PST
(
hide
)
Description:
Patch
Filename:
MIME Type:
Creator:
Srinivasan Vijayaraghavan
Created:
2017-02-02 16:27:51 PST
Size:
77.84 KB
patch
obsolete
>Subversion Revision: 206377 >diff --git a/Tools/ChangeLog b/Tools/ChangeLog >index 9b756a7f616e344b1abac61dfed2a1eb86f7e44d..84c46915558c505a8e6345a226ea89b8c3159211 100644 >--- a/Tools/ChangeLog >+++ b/Tools/ChangeLog >@@ -1,3 +1,117 @@ >+2017-01-30 Srinivasan Vijayaraghavan <svijayaraghavan@apple.com> >+ >+ EWS should run JavaScriptCore tests >+ https://bugs.webkit.org/show_bug.cgi?id=162458 >+ >+ Reviewed by NOBODY (OOPS!). >+ >+ * Scripts/webkitpy/common/checkout/scm/scm_mock.py: >+ (MockSCM.__init__): Add _mockChangedFiles variable. >+ (MockSCM.changed_files): Change to use _mockChangedFiles variables. >+ * Scripts/webkitpy/common/config/ews.json: Add config info for JSC EWS. >+ * Scripts/webkitpy/common/config/ports.py: Add support for JSC EWS in Mac Port. >+ (DeprecatedPort.build_jsc_command): Added - command to build JSC only (quicker than building all of WebKit). >+ (DeprecatedPort.run_javascriptcore_tests_command): Allow JSC EWS to only run JSC tests. >+ (MacPort.run_webkit_tests_command): Check for JSC. >+ * Scripts/webkitpy/common/config/ports_mock.py: >+ (MockPort.run_javascriptcore_tests_command): Add build_style argument. >+ * Scripts/webkitpy/common/config/ports_unittest.py: >+ (DeprecatedPortTest.test_mac_port): Add unit tests for build-jsc shell command. >+ * Scripts/webkitpy/common/net/jsctestresults.py: Added. >+ (JSCTestResults): Added. >+ (JSCTestResults.intersection): Return a JSCTestResults object with failures common to both input objects. >+ (JSCTestResults.results_from_string): Use json library to parse results. >+ (JSCTestResults.__init__): Initialize members. >+ (JSCTestResults.equals): This enables unit testing. >+ (JSCTestResults.is_subset): Checks if one set of failures is a subset of another. >+ (JSCTestResults.stress_test_failures): Getter. >+ (JSCTestResults.all_api_tests_passed): Getter. >+ (JSCTestResults.all_passed): Getter. >+ (JSCTestResults.failing_test_results): Getter. Some of these exist for interopability reasons. >+ (JSCTestResults.test_results): Getter. >+ (JSCTestResults.failing_tests): Getter. >+ (JSCTestResults.did_exceed_test_failure_limit): Getter. >+ * Scripts/webkitpy/common/net/jsctestresults_unittest.py: Added. >+ (JSCTestResultsTest): Class to test JSCTestResults. >+ (JSCTestResultsTest.test_results_from_string): Tests parsing. >+ (JSCTestResultsTest.test_intersection_api_tests): Unit test for intersection() class method. >+ (JSCTestResultsTest.test_intersection_stress_tests): Unit test for intersection() class method. >+ (JSCTestResultsTest.test_intersection_general_case): Unit test for intersection() class method. >+ * Scripts/webkitpy/tool/bot/earlywarningsystemtask.py: >+ (EarlyWarningSystemTask.run): Add check_patch_relevance step. >+ * Scripts/webkitpy/tool/bot/jscews_unittest.py: Added. >+ (MockPatchAnalysisTask): Mocked-out version of PatchAnalysisTask that doesn't run commands. >+ (MockPatchAnalysisTask.__init__): Sets attributes. >+ (MockPatchAnalysisTask._test): Override to not run command. >+ (MockPatchAnalysisTask._build_and_test_without_patch): Override to not run command. >+ (MockPatchAnalysisTask.validate): Assume mocked patch is valid for purposes of testing retry logic. >+ (MockPatchAnalysisTask.test_run_count): Specific to the mocked version, to test retry logic. >+ (MockJSCEarlyWarningSystem): Mocked-out version of AbstarctEarlyWarningSystem so we can provide test results. >+ (MockJSCEarlyWarningSystem.__init__): Sets attributes, also sets group to jsc. >+ (MockJSCEarlyWarningSystem.test_results): Returns test results provided by us, instead of using a JSON reader. >+ (JSCEarlyWarningSystemTest): Class to test retry logic in below situations. >+ (JSCEarlyWarningSystemTest._create_task): Helper function to abstract out common code. >+ (JSCEarlyWarningSystemTest._results_indicate_all_passed): False if input is None or has failures, else True. >+ (JSCEarlyWarningSystemTest.test_success_case): Clean patch on clean tree. >+ (JSCEarlyWarningSystemTest.test_test_failure): Failed patch on clean tree. >+ (JSCEarlyWarningSystemTest.test_fix): Patch that fixes all tree redness. >+ (JSCEarlyWarningSystemTest.test_ineffective_patch): Patch that has same failures as tree. >+ (JSCEarlyWarningSystemTest.test_partially_effective_patch): Patch fixes some failures but adds no new failures. >+ (JSCEarlyWarningSystemTest.test_different_test_failures_in_patch_and_tree): Patch has some failures not in tree. >+ (JSCEarlyWarningSystemTest.test_first_results_could_not_be_read): Patch results not readable. >+ (JSCEarlyWarningSystemTest.test_second_results_could_not_be_read): Patch results not readable on second run. >+ (JSCEarlyWarningSystemTest.test_clean_results_could_not_be_read): Results from clean tree not readable. >+ (JSCEarlyWarningSystemTest.test_flaky_results_on_clean_tree_pass): Patch has one flake and no failures. >+ (JSCEarlyWarningSystemTest.test_flaky_results_on_clean_tree_pass_v2): Patch has one flake and no failures. >+ (JSCEarlyWarningSystemTest.test_flaky_results_on_clean_tree_failure): Patch has flakes and failed tests. >+ (JSCEarlyWarningSystemTest.test_flaky_results_on_red_tree_pass): Patch has same failures as tree, plus a flake. >+ * Scripts/webkitpy/tool/bot/jsctestresultsreader.py: Added. >+ (JSCTestResultsReader): Reads results file. >+ (JSCTestResultsReader.__init__): Sets attributes. >+ (JSCTestResultsReader._read_file_contents): Reads file. >+ (JSCTestResultsReader._create_jsc_test_results): Creates a JSCTestResults object. >+ (JSCTestResultsReader.results): Invokes _create_jsc_test_results to return a JSCTestResults object. >+ * Scripts/webkitpy/tool/bot/patchanalysistask.py: >+ (PatchIsNotApplicable): Exception for when patch doesn't have relevant changes. >+ (PatchAnalysisTask._check_patch_relevance): Added. >+ (PatchAnalysisTask._build): Check for JSC. >+ (PatchAnalysisTask._build_without_patch): Check for JSC. >+ (PatchAnalysisTask._test): Check for JSC. >+ (PatchAnalysisTask._build_and_test_without_patch): Check for JSC. >+ (PatchAnalysisTask._test_patch): Add retry logic for JSC. >+ * Scripts/webkitpy/tool/commands/download.py: >+ (CheckPatchRelevance): Add check-patch-relevance command. >+ * Scripts/webkitpy/tool/commands/earlywarningsystem.py: >+ (AbstractEarlyWarningSystem.begin_work_queue): Use JSCTestResultsReader not LayoutTestResultsReader in JSC EWS. >+ (AbstractEarlyWarningSystem.review_patch): Handle PatchIsApplicable >+ (AbstractEarlyWarningSystem.test_results): _layout_test_results_reader -> _test_results_reader. >+ (AbstractEarlyWarningSystem.archive_last_test_results): _layout_test_results_reader -> _test_results_reader. >+ (AbstractEarlyWarningSystem.group): This attribute determines the type of EWS (eg. JSC). >+ (AbstractEarlyWarningSystem.load_ews_classes): Add _group. >+ * Scripts/webkitpy/tool/commands/earlywarningsystem_unittest.py: >+ (AbstractEarlyWarningSystemTest.test_failing_tests_message.TestEWS): Add _group. >+ (AbstractEarlyWarningSystemTest.test_failing_jsc_tests_message.TestEWS): Added test for jsc failures message. >+ (AbstractEarlyWarningSystemTest): Add _group variable. >+ (EarlyWarningSystemTest._default_expected_logs): Add check-patch-relevance step and --group option. >+ * Scripts/webkitpy/tool/steps/__init__.py: Add CheckPatchRelevance import. >+ * Scripts/webkitpy/tool/steps/build.py: >+ (Build.options): Check for JSC. >+ (Build.build): Check for JSC. >+ (Build.run): Check for JSC. >+ * Scripts/webkitpy/tool/steps/checkpatchrelevance.py: Added. >+ (CheckPatchRelevance): Added. >+ (CheckPatchRelevance._does_contain_change_in_paths): Abstract function to perform regex matching. >+ (CheckPatchRelevance.run): Check if changed files in patch belong in certain folders. >+ * Scripts/webkitpy/tool/steps/options.py: >+ (Options): Add --group command line option. >+ * Scripts/webkitpy/tool/steps/runtests.py: >+ (RunTests.options): Add group. >+ (RunTests.run): Check for JSC. >+ (RunTests._run_webkit_tests): Check for JSC. >+ (RunTests): Add _group attribute. >+ (RunTests._run_javascriptcore_tests): New. >+ * Scripts/webkitpy/tool/steps/steps_unittest.py: Unit tests. >+ > 2016-09-25 Gyuyoung Kim <gyuyoung.kim@webkit.org> > > [EFL] Adjust EFL indentation to EFL MiniBrowser >diff --git a/Tools/Scripts/webkitpy/common/checkout/scm/scm_mock.py b/Tools/Scripts/webkitpy/common/checkout/scm/scm_mock.py >index c5d10fcb189b0a8b519fa55625022a70c486d44b..9774ec8ae873a1b0b8376071806679215809eeec 100644 >--- a/Tools/Scripts/webkitpy/common/checkout/scm/scm_mock.py >+++ b/Tools/Scripts/webkitpy/common/checkout/scm/scm_mock.py >@@ -1,4 +1,5 @@ > # Copyright (C) 2011 Google Inc. All rights reserved. >+# Copyright (C) 2017 Apple Inc. All rights reserved. > # > # Redistribution and use in source and binary forms, with or without > # modification, are permitted provided that the following conditions are >@@ -37,6 +38,7 @@ class MockSCM(object): > self.added_paths = set() > self._filesystem = filesystem or MockFileSystem() > self._executive = executive or MockExecutive() >+ self._mockChangedFiles = ["MockFile1"] > > def add(self, destination_path): > self.add_list([destination_path]) >@@ -71,7 +73,7 @@ class MockSCM(object): > return self._filesystem.join(self.checkout_root, *comps) > > def changed_files(self, git_commit=None): >- return ["MockFile1"] >+ return self._mockChangedFiles > > def changed_files_for_revision(self, revision): > return ["MockFile1"] >diff --git a/Tools/Scripts/webkitpy/common/config/ews.json b/Tools/Scripts/webkitpy/common/config/ews.json >index 35d4985ab0f93d127b3721e4601cd6d89dfccf3c..e5340fe5c3c21ae81ad1ef7e03f7fb81f3839cff 100644 >--- a/Tools/Scripts/webkitpy/common/config/ews.json >+++ b/Tools/Scripts/webkitpy/common/config/ews.json >@@ -50,5 +50,11 @@ > "port": "mac", > "name": "mac-32bit-ews", > "architecture": "i386" >+ }, >+ "JSC EWS": { >+ "port": "mac", >+ "name": "jsc-ews", >+ "group": "jsc", >+ "runTests": true > } > } >diff --git a/Tools/Scripts/webkitpy/common/config/ports.py b/Tools/Scripts/webkitpy/common/config/ports.py >index a0d50dcb6ea0b5fde0b3c17ef159ceafee1a2e98..d37ed9d1c5e2550e48834e7ef00bae4c24cbd784 100644 >--- a/Tools/Scripts/webkitpy/common/config/ports.py >+++ b/Tools/Scripts/webkitpy/common/config/ports.py >@@ -1,6 +1,6 @@ > # Copyright (C) 2009, Google Inc. All rights reserved. > # Copyright (C) 2013 Nokia Corporation and/or its subsidiary(-ies). >-# Copyright (C) 2015 Apple Inc. All rights reserved. >+# Copyright (C) 2015, 2017 Apple Inc. All rights reserved. > # > # Redistribution and use in source and binary forms, with or without > # modification, are permitted provided that the following conditions are >@@ -103,8 +103,21 @@ class DeprecatedPort(object): > command.append("--release") > return command > >- def run_javascriptcore_tests_command(self): >- return self.script_shell_command("run-javascriptcore-tests") >+ def build_jsc_command(self, build_style=None): >+ command = self.script_shell_command("build-jsc") >+ if build_style == "debug": >+ command.append("--debug") >+ if build_style == "release": >+ command.append("--release") >+ return command >+ >+ def run_javascriptcore_tests_command(self, build_style=None): >+ command = self.script_shell_command("run-javascriptcore-tests") >+ command.append("--no-fail-fast") >+ command.append("--json-output=jsc_test_results.json") >+ if (build_style == "debug"): >+ command.append("--debug") >+ return command > > def run_webkit_tests_command(self, build_style=None): > command = self.script_shell_command("run-webkit-tests") >@@ -150,7 +163,10 @@ class IOSSimulatorWK2Port(DeprecatedPort): > class MacPort(DeprecatedPort): > port_flag_name = "mac" > >- def run_webkit_tests_command(self, build_style=None): >+ def run_webkit_tests_command(self, build_style=None, test_suite=None): >+ if (test_suite == "jsc"): >+ return super(MacPort, self).run_javascriptcore_tests_command(build_style) >+ > command = super(MacPort, self).run_webkit_tests_command(build_style) > command.append("--dump-render-tree") > return command >diff --git a/Tools/Scripts/webkitpy/common/config/ports_mock.py b/Tools/Scripts/webkitpy/common/config/ports_mock.py >index 61743debb4ebee3756ba7c20f014823540d2dca6..ca1b18a31068423bdce07cdeb7d7d7d199f400ab 100644 >--- a/Tools/Scripts/webkitpy/common/config/ports_mock.py >+++ b/Tools/Scripts/webkitpy/common/config/ports_mock.py >@@ -1,4 +1,5 @@ > # Copyright (C) 2011 Google Inc. All rights reserved. >+# Copyright (C) 2017 Apple Inc. All rights reserved. > # > # Redistribution and use in source and binary forms, with or without > # modification, are permitted provided that the following conditions are >@@ -49,7 +50,7 @@ class MockPort(object): > def run_perl_unittests_command(self): > return ['mock-test-webkitperl'] > >- def run_javascriptcore_tests_command(self): >+ def run_javascriptcore_tests_command(self, build_style=None): > return ['mock-run-javacriptcore-tests'] > > def run_webkit_tests_command(self, build_style=None): >diff --git a/Tools/Scripts/webkitpy/common/config/ports_unittest.py b/Tools/Scripts/webkitpy/common/config/ports_unittest.py >index 08d545236da3703e928f1ceed1dd9102d9b8622e..16cd7f928251bbc996cb646e4359d73fdb535b26 100644 >--- a/Tools/Scripts/webkitpy/common/config/ports_unittest.py >+++ b/Tools/Scripts/webkitpy/common/config/ports_unittest.py >@@ -1,4 +1,5 @@ > # Copyright (c) 2009, Google Inc. All rights reserved. >+# Copyright (C) 2017 Apple Inc. All rights reserved. > # > # Redistribution and use in source and binary forms, with or without > # modification, are permitted provided that the following conditions are >@@ -39,6 +40,10 @@ class DeprecatedPortTest(unittest.TestCase): > self.assertEqual(MacPort().build_webkit_command(build_style="debug"), DeprecatedPort().script_shell_command("build-webkit") + ["--debug"]) > self.assertEqual(MacPort().build_webkit_command(build_style="release"), DeprecatedPort().script_shell_command("build-webkit") + ["--release"]) > >+ self.assertEqual(MacPort().build_jsc_command(), DeprecatedPort().script_shell_command("build-jsc")) >+ self.assertEqual(MacPort().build_jsc_command(build_style="release"), DeprecatedPort().script_shell_command("build-jsc") + ["--release"]) >+ self.assertEqual(MacPort().build_jsc_command(build_style="debug"), DeprecatedPort().script_shell_command("build-jsc") + ["--debug"]) >+ > def test_gtk_wk2_port(self): > self.assertEqual(GtkWK2Port().flag(), "--port=gtk-wk2") > self.assertEqual(GtkWK2Port().run_webkit_tests_command(), DeprecatedPort().script_shell_command("run-webkit-tests") + ["--gtk"]) >diff --git a/Tools/Scripts/webkitpy/common/net/jsctestresults.py b/Tools/Scripts/webkitpy/common/net/jsctestresults.py >new file mode 100644 >index 0000000000000000000000000000000000000000..be6eb0e30f6736ba9942cf0f8c168d32132aa7b5 >--- /dev/null >+++ b/Tools/Scripts/webkitpy/common/net/jsctestresults.py >@@ -0,0 +1,86 @@ >+# Copyright (C) 2017 Apple Inc. All rights reserved. >+# >+# Redistribution and use in source and binary forms, with or without >+# modification, are permitted provided that the following conditions >+# are met: >+# 1. Redistributions of source code must retain the above copyright >+# notice, this list of conditions and the following disclaimer. >+# 2. Redistributions in binary form must reproduce the above copyright >+# notice, this list of conditions and the following disclaimer in the >+# documentation and/or other materials provided with the distribution. >+# >+# THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS'' AND >+# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED >+# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE >+# DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS BE LIABLE FOR >+# ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL >+# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR >+# SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER >+# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, >+# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE >+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. >+ >+import json >+import logging >+ >+_log = logging.getLogger(__name__) >+ >+ >+class JSCTestResults(object): >+ @classmethod >+ def intersection(cls, first, second): >+ intersection_api_tests_passed = first._all_api_tests_passed or second._all_api_tests_passed >+ intersection_stress_test_failures = list(set(first._stress_test_failures) & set(second._stress_test_failures)) >+ return cls(intersection_api_tests_passed, intersection_stress_test_failures) >+ >+ @classmethod >+ def results_from_string(cls, string): >+ if not string: >+ return None >+ >+ try: >+ parsed_results = json.loads(string) >+ except ValueError: >+ return None >+ >+ if 'allApiTestsPassed' not in parsed_results or 'stressTestFailures' not in parsed_results: >+ return None >+ >+ return cls(parsed_results['allApiTestsPassed'], parsed_results['stressTestFailures']) >+ >+ def __init__(self, all_api_tests_passed, stress_test_failures): >+ self._all_api_tests_passed = all_api_tests_passed >+ self._stress_test_failures = stress_test_failures >+ >+ self._test_results = stress_test_failures[:] >+ if not self._all_api_tests_passed: >+ self._test_results.append("apiTests") >+ >+ def equals(self, other): >+ return self._all_api_tests_passed == other._all_api_tests_passed and \ >+ set(self._stress_test_failures) == set(other._stress_test_failures) >+ >+ def is_subset(self, other): >+ return set(self.test_results()) <= set(other.test_results()) >+ >+ def stress_test_failures(self): >+ return self._stress_test_failures >+ >+ def all_api_tests_passed(self): >+ return self._all_api_tests_passed >+ >+ def all_passed(self): >+ return self._all_api_tests_passed and not self._stress_test_failures >+ >+ def failing_test_results(self): >+ return self.stress_test_failures() >+ >+ def test_results(self): >+ return self._test_results >+ >+ def failing_tests(self): >+ return self._test_results >+ >+ # No defined failure limit for JSC. >+ def did_exceed_test_failure_limit(self): >+ return False >diff --git a/Tools/Scripts/webkitpy/common/net/jsctestresults_unittest.py b/Tools/Scripts/webkitpy/common/net/jsctestresults_unittest.py >new file mode 100644 >index 0000000000000000000000000000000000000000..18ef444a568e460880459176c5ffac06f15f17b6 >--- /dev/null >+++ b/Tools/Scripts/webkitpy/common/net/jsctestresults_unittest.py >@@ -0,0 +1,75 @@ >+# Copyright (C) 2017 Apple Inc. All rights reserved. >+# >+# Redistribution and use in source and binary forms, with or without >+# modification, are permitted provided that the following conditions >+# are met: >+# 1. Redistributions of source code must retain the above copyright >+# notice, this list of conditions and the following disclaimer. >+# 2. Redistributions in binary form must reproduce the above copyright >+# notice, this list of conditions and the following disclaimer in the >+# documentation and/or other materials provided with the distribution. >+# >+# THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS'' AND >+# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED >+# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE >+# DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS BE LIABLE FOR >+# ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL >+# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR >+# SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER >+# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, >+# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE >+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. >+ >+import unittest >+ >+from webkitpy.common.net.jsctestresults import JSCTestResults >+ >+ >+class JSCTestResultsTest(unittest.TestCase): >+ def test_results_from_string(self): >+ none_item = None >+ empty_string = "" >+ invalid_json = '{"allApiTestsPassed":' >+ incomplete_json_v1 = '{"allApiTestsPassed": true}' >+ incomplete_json_v2 = '{"stressTestFailures":[]}' >+ self.assertEqual(None, JSCTestResults.results_from_string(none_item)) >+ self.assertEqual(None, JSCTestResults.results_from_string(empty_string)) >+ self.assertEqual(None, JSCTestResults.results_from_string(invalid_json)) >+ self.assertEqual(None, JSCTestResults.results_from_string(incomplete_json_v1)) >+ self.assertEqual(None, JSCTestResults.results_from_string(incomplete_json_v2)) >+ >+ no_failures_string = '{"allApiTestsPassed": true, "stressTestFailures":[]}' >+ no_failures_results = JSCTestResults(True, []) >+ self.assertTrue(no_failures_results.equals(JSCTestResults.results_from_string(no_failures_string))) >+ >+ api_test_failures_string = '{"allApiTestsPassed": false, "stressTestFailures":[]}' >+ api_test_failures_results = JSCTestResults(False, []) >+ self.assertTrue(api_test_failures_results.equals(JSCTestResults.results_from_string(api_test_failures_string))) >+ >+ many_failures_string = '{"allApiTestsPassed": false, "stressTestFailures":["es6.yaml/es6/typed_arrays_Int16Array.js.default", "es6.yaml/es6/typed_arrays_Int8Array.js.default"]}' >+ many_failures_results = JSCTestResults(False, ["es6.yaml/es6/typed_arrays_Int16Array.js.default", "es6.yaml/es6/typed_arrays_Int8Array.js.default"]) >+ self.assertTrue(many_failures_results.equals(JSCTestResults.results_from_string(many_failures_string))) >+ >+ self.assertFalse(no_failures_results == api_test_failures_results) >+ self.assertFalse(api_test_failures_results == many_failures_results) >+ >+ def test_intersection_api_tests(self): >+ results1 = JSCTestResults(False, []) >+ results2 = JSCTestResults(True, []) >+ >+ expected_intersection = JSCTestResults(True, []) >+ self.assertTrue(expected_intersection.equals(JSCTestResults.intersection(results1, results2))) >+ >+ def test_intersection_stress_tests(self): >+ results1 = JSCTestResults(True, ['failure1', 'failure2']) >+ results2 = JSCTestResults(True, ['failure1', 'failure3']) >+ >+ expected_intersection = JSCTestResults(True, ['failure1']) >+ self.assertTrue(expected_intersection.equals(JSCTestResults.intersection(results1, results2))) >+ >+ def test_intersection_general_case(self): >+ results1 = JSCTestResults(True, ['failure1', 'failure2']) >+ results2 = JSCTestResults(False, ['failure1']) >+ >+ expected_intersection = JSCTestResults(True, ['failure1']) >+ self.assertTrue(expected_intersection.equals(JSCTestResults.intersection(results1, results2))) >diff --git a/Tools/Scripts/webkitpy/tool/bot/earlywarningsystemtask.py b/Tools/Scripts/webkitpy/tool/bot/earlywarningsystemtask.py >index fc308b43d1d172ab73b0e3af80d1b139dd3c1032..ecfacd011be883b4bb1e5c7d7dffdbdcb459de35 100644 >--- a/Tools/Scripts/webkitpy/tool/bot/earlywarningsystemtask.py >+++ b/Tools/Scripts/webkitpy/tool/bot/earlywarningsystemtask.py >@@ -1,4 +1,5 @@ > # Copyright (c) 2011 Google Inc. All rights reserved. >+# Copyright (C) 2017 Apple Inc. All rights reserved. > # > # Redistribution and use in source and binary forms, with or without > # modification, are permitted provided that the following conditions are >@@ -26,7 +27,7 @@ > # (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE > # OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. > >-from webkitpy.tool.bot.patchanalysistask import PatchAnalysisTask, PatchAnalysisTaskDelegate, UnableToApplyPatch, PatchIsNotValid >+from webkitpy.tool.bot.patchanalysistask import PatchAnalysisTask, PatchAnalysisTaskDelegate, UnableToApplyPatch, PatchIsNotValid, PatchIsNotApplicable > > > class EarlyWarningSystemTaskDelegate(PatchAnalysisTaskDelegate): >@@ -58,6 +59,8 @@ class EarlyWarningSystemTask(PatchAnalysisTask): > return False > if not self._apply(): > raise UnableToApplyPatch(self._patch) >+ if not self._check_patch_relevance(): >+ raise PatchIsNotApplicable(self._patch) > if not self._build(): > if not self._build_without_patch(): > return False >diff --git a/Tools/Scripts/webkitpy/tool/bot/jscews_unittest.py b/Tools/Scripts/webkitpy/tool/bot/jscews_unittest.py >new file mode 100644 >index 0000000000000000000000000000000000000000..8713000256c3fdc073518b2b4953dfe90f31cb09 >--- /dev/null >+++ b/Tools/Scripts/webkitpy/tool/bot/jscews_unittest.py >@@ -0,0 +1,219 @@ >+# Copyright (C) 2017 Apple Inc. All rights reserved. >+# >+# Redistribution and use in source and binary forms, with or without >+# modification, are permitted provided that the following conditions >+# are met: >+# 1. Redistributions of source code must retain the above copyright >+# notice, this list of conditions and the following disclaimer. >+# 2. Redistributions in binary form must reproduce the above copyright >+# notice, this list of conditions and the following disclaimer in the >+# documentation and/or other materials provided with the distribution. >+# >+# THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS'' AND >+# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED >+# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE >+# DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS BE LIABLE FOR >+# ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL >+# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR >+# SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER >+# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, >+# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE >+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. >+ >+import logging >+import unittest >+ >+from webkitpy.common.net.jsctestresults import JSCTestResults >+from webkitpy.common.system.executive import ScriptError >+from webkitpy.tool.bot.patchanalysistask import * >+from webkitpy.tool.commands.earlywarningsystem import AbstractEarlyWarningSystem >+from webkitpy.tool.mocktool import MockTool >+ >+_log = logging.getLogger(__name__) >+ >+ >+class MockPatchAnalysisTask(PatchAnalysisTask): >+ def __init__(self, delegate, patch, patches_passed_all_tests): >+ self._delegate = delegate >+ self._patch = patch >+ self._script_error = None >+ self._results_archive_from_patch_test_run = None >+ self._results_from_patch_test_run = None >+ self.error = None >+ self._patches_passed_all_tests = patches_passed_all_tests >+ self._test_run_count = 0 >+ self.failure_status_id = 42 >+ >+ def _test(self): >+ self._test_run_count += 1 >+ >+ if self._patches_passed_all_tests.pop() == True: >+ return True >+ else: >+ self._script_error = ScriptError("Regression test") >+ return False >+ >+ def _build_and_test_without_patch(self): >+ self._test_run_count += 1 >+ return True >+ >+ def validate(self): >+ return True >+ >+ def test_run_count(self): >+ return self._test_run_count >+ >+ >+# This is the delegate to be used with MockPatchAnalysisTask, above. >+class MockJSCEarlyWarningSystem(AbstractEarlyWarningSystem): >+ def __init__(self, first_test_results, second_test_results, clean_test_results): >+ AbstractEarlyWarningSystem.__init__(self) >+ self._group = "jsc" >+ self._results_in_order = [clean_test_results, second_test_results, first_test_results] >+ >+ def test_results(self): >+ return self._results_in_order.pop() >+ >+ >+class JSCEarlyWarningSystemTest(unittest.TestCase): >+ def _results_indicate_all_passed(self, results): >+ if results == None: >+ return False >+ return results.all_passed() >+ >+ def _create_task(self, first_test_results, second_test_results, clean_test_results): >+ queue = MockJSCEarlyWarningSystem(first_test_results, second_test_results, clean_test_results) >+ tool = MockTool(log_executive=True) >+ patch = tool.bugs.fetch_attachment(10000) >+ patches_passed_all_tests = map(self._results_indicate_all_passed, [second_test_results, first_test_results]) >+ return MockPatchAnalysisTask(queue, patch, patches_passed_all_tests) >+ >+ def test_success_case(self): >+ first_test_results = JSCTestResults(True, []) >+ second_test_results = JSCTestResults(True, []) >+ clean_test_results = JSCTestResults(True, []) >+ task = self._create_task(first_test_results, second_test_results, clean_test_results) >+ >+ return_value = task._test_patch() >+ self.assertEqual(task.test_run_count(), 1) >+ self.assertEqual(return_value, True) >+ >+ def test_test_failure(self): >+ first_test_results = JSCTestResults(True, ['Fail.js']) >+ second_test_results = JSCTestResults(True, ['Fail.js']) >+ clean_test_results = JSCTestResults(True, []) >+ task = self._create_task(first_test_results, second_test_results, clean_test_results) >+ >+ with self.assertRaises(ScriptError): >+ return_value = task._test_patch() >+ self.assertEqual(task.test_run_count(), 3) >+ >+ def test_fix(self): >+ first_test_results = JSCTestResults(True, []) >+ second_test_results = JSCTestResults(True, []) >+ clean_test_results = JSCTestResults(True, ['Fail.js']) >+ task = self._create_task(first_test_results, second_test_results, clean_test_results) >+ >+ return_value = task._test_patch() >+ self.assertEqual(task.test_run_count(), 1) >+ self.assertEqual(return_value, True) >+ >+ def test_ineffective_patch(self): >+ first_test_results = JSCTestResults(False, ['failure1.js', 'failure2.js']) >+ second_test_results = JSCTestResults(False, ['failure1.js', 'failure2.js']) >+ clean_test_results = JSCTestResults(False, ['failure1.js', 'failure2.js']) >+ task = self._create_task(first_test_results, second_test_results, clean_test_results) >+ >+ return_value = task._test_patch() >+ self.assertEqual(task.test_run_count(), 3) >+ self.assertEqual(return_value, True) >+ >+ def test_partially_effective_patch(self): >+ first_test_results = JSCTestResults(True, ['failure2.js']) >+ second_test_results = JSCTestResults(True, ['failure2.js']) >+ clean_test_results = JSCTestResults(False, ['failure1.js', 'failure2.js']) >+ task = self._create_task(first_test_results, second_test_results, clean_test_results) >+ >+ return_value = task._test_patch() >+ self.assertEqual(task.test_run_count(), 3) >+ self.assertEqual(return_value, True) >+ >+ def test_different_test_failures_in_patch_and_tree(self): >+ first_test_results = JSCTestResults(False, []) >+ second_test_results = JSCTestResults(False, []) >+ clean_test_results = JSCTestResults(True, ['failure1.js', 'failure2.js']) >+ task = self._create_task(first_test_results, second_test_results, clean_test_results) >+ >+ with self.assertRaises(ScriptError): >+ return_value = task._test_patch() >+ self.assertEqual(task.test_run_count(), 3) >+ >+ def test_first_results_could_not_be_read(self): >+ first_test_results = None >+ second_test_results = JSCTestResults(True, []) >+ clean_test_results = JSCTestResults(True, []) >+ task = self._create_task(first_test_results, second_test_results, clean_test_results) >+ >+ with self.assertRaises(ScriptError): >+ return_value = task._test_patch() >+ self.assertEqual(task.test_run_count(), 1) >+ >+ def test_second_results_could_not_be_read(self): >+ first_test_results = JSCTestResults(False, ['failure1.js', 'failure2.js']) >+ second_test_results = None >+ clean_test_results = JSCTestResults(True, []) >+ task = self._create_task(first_test_results, second_test_results, clean_test_results) >+ >+ with self.assertRaises(ScriptError): >+ return_value = task._test_patch() >+ self.assertEqual(task.test_run_count(), 2) >+ >+ def test_clean_results_could_not_be_read(self): >+ first_test_results = JSCTestResults(True, ['failure2.js']) >+ second_test_results = JSCTestResults(True, ['failure2.js']) >+ clean_test_results = None >+ task = self._create_task(first_test_results, second_test_results, clean_test_results) >+ >+ with self.assertRaises(ScriptError): >+ return_value = task._test_patch() >+ self.assertEqual(task.test_run_count(), 3) >+ >+ def test_flaky_results_on_clean_tree_pass(self): >+ first_test_results = JSCTestResults(True, ['failure2.js']) >+ second_test_results = JSCTestResults(True, []) >+ clean_test_results = JSCTestResults(True, []) >+ task = self._create_task(first_test_results, second_test_results, clean_test_results) >+ >+ return_value = task._test_patch() >+ self.assertEqual(return_value, True) >+ self.assertEqual(task.test_run_count(), 2) >+ >+ def test_flaky_results_on_clean_tree_pass_v2(self): >+ first_test_results = JSCTestResults(True, []) >+ second_test_results = JSCTestResults(True, ['failure2.js']) >+ clean_test_results = JSCTestResults(True, []) >+ task = self._create_task(first_test_results, second_test_results, clean_test_results) >+ >+ return_value = task._test_patch() >+ self.assertEqual(return_value, True) >+ self.assertEqual(task.test_run_count(), 1) >+ >+ def test_flaky_results_on_clean_tree_failure(self): >+ first_test_results = JSCTestResults(False, ['failure1.js', 'failure2.js']) >+ second_test_results = JSCTestResults(True, ['failure2.js']) >+ clean_test_results = JSCTestResults(True, []) >+ task = self._create_task(first_test_results, second_test_results, clean_test_results) >+ >+ with self.assertRaises(ScriptError): >+ return_value = task._test_patch() >+ self.assertEqual(task.test_run_count(), 3) >+ >+ def test_flaky_results_on_red_tree_pass(self): >+ first_test_results = JSCTestResults(True, ['failure1.js']) >+ second_test_results = JSCTestResults(True, ['failure1.js', 'failure2.js']) >+ clean_test_results = JSCTestResults(True, ['failure1.js']) >+ task = self._create_task(first_test_results, second_test_results, clean_test_results) >+ >+ return_value = task._test_patch() >+ self.assertEqual(return_value, True) >+ self.assertEqual(task.test_run_count(), 3) >diff --git a/Tools/Scripts/webkitpy/tool/bot/jsctestresultsreader.py b/Tools/Scripts/webkitpy/tool/bot/jsctestresultsreader.py >new file mode 100644 >index 0000000000000000000000000000000000000000..8ac532b19e7ea55d35b9139b4f2099805ba0049e >--- /dev/null >+++ b/Tools/Scripts/webkitpy/tool/bot/jsctestresultsreader.py >@@ -0,0 +1,47 @@ >+# Copyright (C) 2017 Apple Inc. All rights reserved. >+# >+# Redistribution and use in source and binary forms, with or without >+# modification, are permitted provided that the following conditions >+# are met: >+# 1. Redistributions of source code must retain the above copyright >+# notice, this list of conditions and the following disclaimer. >+# 2. Redistributions in binary form must reproduce the above copyright >+# notice, this list of conditions and the following disclaimer in the >+# documentation and/or other materials provided with the distribution. >+# >+# THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS'' AND >+# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED >+# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE >+# DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS BE LIABLE FOR >+# ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL >+# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR >+# SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER >+# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, >+# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE >+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. >+ >+import logging >+ >+from webkitpy.common.net.jsctestresults import JSCTestResults >+ >+_log = logging.getLogger(__name__) >+ >+ >+class JSCTestResultsReader(object): >+ def __init__(self, host, results_directory): >+ self._host = host >+ self._results_directory = results_directory >+ >+ def _read_file_contents(self, path): >+ try: >+ return self._host.filesystem.read_text_file(path) >+ except IOError, e: # File does not exist or can't be read. >+ return None >+ >+ def _create_jsc_test_results(self): >+ results_path = self._host.filesystem.join(self._results_directory, "jsc_test_results.json") >+ contents = self._read_file_contents(results_path) >+ return JSCTestResults.results_from_string(contents) >+ >+ def results(self): >+ return self._create_jsc_test_results() >diff --git a/Tools/Scripts/webkitpy/tool/bot/patchanalysistask.py b/Tools/Scripts/webkitpy/tool/bot/patchanalysistask.py >index 752316416c5f9d0a3e59ef6d05e31785f9174629..a740fd68415c3ad7d3988103cb4a3538147cb01b 100644 >--- a/Tools/Scripts/webkitpy/tool/bot/patchanalysistask.py >+++ b/Tools/Scripts/webkitpy/tool/bot/patchanalysistask.py >@@ -1,4 +1,5 @@ > # Copyright (c) 2010 Google Inc. All rights reserved. >+# Copyright (C) 2017 Apple Inc. All rights reserved. > # > # Redistribution and use in source and binary forms, with or without > # modification, are permitted provided that the following conditions are >@@ -28,6 +29,7 @@ > > from webkitpy.common.system.executive import ScriptError > from webkitpy.common.net.layouttestresults import LayoutTestResults >+from webkitpy.common.net.jsctestresults import JSCTestResults > > > class UnableToApplyPatch(Exception): >@@ -43,6 +45,11 @@ class PatchIsNotValid(Exception): > self.failure_message = failure_message > > >+class PatchIsNotApplicable(Exception): >+ def __init__(self, patch): >+ Exception.__init__(self) >+ self.patch = patch >+ > class PatchAnalysisTaskDelegate(object): > def parent_command(self): > raise NotImplementedError("subclasses must implement") >@@ -122,28 +129,53 @@ class PatchAnalysisTask(object): > "Applied patch", > "Patch does not apply") > >+ def _check_patch_relevance(self): >+ args = [ >+ "check-patch-relevance", >+ ] >+ >+ if hasattr(self._delegate, 'group'): >+ args.append("--group=%s" % self._delegate.group()) >+ >+ return self._run_command( >+ args, >+ "Checked relevance of patch", >+ "Patch was not relevant") >+ > def _build(self): >- return self._run_command([ >+ args = [ > "build", > "--no-clean", > "--no-update", > "--build-style=%s" % self._delegate.build_style(), >- ], >- "Built patch", >- "Patch does not build") >+ ] >+ >+ if hasattr(self._delegate, 'group'): >+ args.append("--group=%s" % self._delegate.group()) >+ >+ return self._run_command( >+ args, >+ "Built patch", >+ "Patch does not build") > > def _build_without_patch(self): >- return self._run_command([ >+ args = [ > "build", > "--force-clean", > "--no-update", > "--build-style=%s" % self._delegate.build_style(), >- ], >- "Able to build without patch", >- "Unable to build without patch") >+ ] >+ >+ if hasattr(self._delegate, 'group'): >+ args.append("--group=%s" % self._delegate.group()) >+ >+ return self._run_command( >+ args, >+ "Able to build without patch", >+ "Unable to build without patch") > > def _test(self): >- return self._run_command([ >+ args = [ > "build-and-test", > "--no-clean", > "--no-update", >@@ -151,12 +183,18 @@ class PatchAnalysisTask(object): > "--test", > "--non-interactive", > "--build-style=%s" % self._delegate.build_style(), >- ], >- "Passed tests", >- "Patch does not pass tests") >+ ] >+ >+ if hasattr(self._delegate, 'group'): >+ args.append("--group=%s" % self._delegate.group()) >+ >+ return self._run_command( >+ args, >+ "Passed tests", >+ "Patch does not pass tests") > > def _build_and_test_without_patch(self): >- return self._run_command([ >+ args = [ > "build-and-test", > "--force-clean", > "--no-update", >@@ -164,9 +202,15 @@ class PatchAnalysisTask(object): > "--test", > "--non-interactive", > "--build-style=%s" % self._delegate.build_style(), >- ], >- "Able to pass tests without patch", >- "Unable to pass tests without patch (tree is red?)") >+ ] >+ >+ if hasattr(self._delegate, 'group'): >+ args.append("--group=%s" % self._delegate.group()) >+ >+ return self._run_command( >+ args, >+ "Able to pass tests without patch", >+ "Unable to pass tests without patch (tree is red?)") > > def _land(self): > # Unclear if this should pass --quiet or not. If --parent-command always does the reporting, then it should. >@@ -210,69 +254,98 @@ class PatchAnalysisTask(object): > if self._test(): > return True > >- # Note: archive_last_test_results deletes the results directory, making these calls order-sensitve. >- # We could remove this dependency by building the test_results from the archive. >- first_results = self._delegate.test_results() >- first_results_archive = self._delegate.archive_last_test_results(self._patch) >- first_script_error = self._script_error >- first_failure_status_id = self.failure_status_id >- >- if self._test() and not first_results.did_exceed_test_failure_limit(): >- # Only report flaky tests if we were successful at parsing results.json and archiving results. >- if first_results and first_results_archive: >- self._report_flaky_tests(first_results.failing_test_results(), first_results_archive) >- return True >+ # FIXME: Abstract out the retry logic instead of casing on the test suite here. >+ if hasattr(self._delegate, 'group') and self._delegate.group() == "jsc": >+ first_results = self._delegate.test_results() >+ first_script_error = self._script_error >+ first_failure_status_id = self.failure_status_id >+ if first_results is None: >+ return self.report_failure(None, None, ScriptError("Could not read JSC test results for patch.")) > >- second_results = self._delegate.test_results() >- second_results_archive = self._delegate.archive_last_test_results(self._patch) >- second_script_error = self._script_error >- second_failure_status_id = self.failure_status_id >+ if self._test(): >+ return True >+ second_results = self._delegate.test_results() >+ second_script_error = self._script_error >+ if second_results is None: >+ return self.report_failure(None, None, ScriptError("Could not read JSC test results on second run.")) >+ >+ consistent_results = JSCTestResults.intersection(first_results, second_results) > >- if second_results.did_exceed_test_failure_limit() and first_results.did_exceed_test_failure_limit(): > self._build_and_test_without_patch() > clean_tree_results = self._delegate.test_results() >+ if clean_tree_results is None: >+ return self.report_failure(None, None, ScriptError("Could not read JSC test results for clean tree.")) > >- if (len(first_results.failing_tests()) - len(clean_tree_results.failing_tests())) <= 5: >- return False >+ if consistent_results.is_subset(clean_tree_results): >+ return True > > self.failure_status_id = first_failure_status_id >+ return self.report_failure(None, consistent_results, first_script_error) > >- return self.report_failure(first_results_archive, first_results, first_script_error) >+ else: >+ # Note: archive_last_test_results deletes the results directory, making these calls order-sensitve. >+ # We could remove this dependency by building the test_results from the archive. >+ first_results = self._delegate.test_results() >+ first_results_archive = self._delegate.archive_last_test_results(self._patch) >+ first_script_error = self._script_error >+ first_failure_status_id = self.failure_status_id > >- if second_results.did_exceed_test_failure_limit(): >- self._should_defer_patch_or_throw(first_results.failing_test_results(), first_results_archive, first_script_error, first_failure_status_id) >- return False >+ if self._test() and not first_results.did_exceed_test_failure_limit(): >+ # Only report flaky tests if we were successful at parsing results.json and archiving results. >+ if first_results and first_results_archive: >+ self._report_flaky_tests(first_results.failing_test_results(), first_results_archive) >+ return True > >- if first_results.did_exceed_test_failure_limit(): >- self._should_defer_patch_or_throw(second_results.failing_test_results(), second_results_archive, second_script_error, second_failure_status_id) >- return False >+ second_results = self._delegate.test_results() >+ second_results_archive = self._delegate.archive_last_test_results(self._patch) >+ second_script_error = self._script_error >+ second_failure_status_id = self.failure_status_id > >- if self._results_failed_different_tests(first_results, second_results): >- first_failing_results_set = frozenset(first_results.failing_test_results()) >- second_failing_results_set = frozenset(second_results.failing_test_results()) >+ if second_results.did_exceed_test_failure_limit() and first_results.did_exceed_test_failure_limit(): >+ self._build_and_test_without_patch() >+ clean_tree_results = self._delegate.test_results() > >- tests_that_only_failed_first = first_failing_results_set.difference(second_failing_results_set) >- self._report_flaky_tests(tests_that_only_failed_first, first_results_archive) >+ if (len(first_results.failing_tests()) - len(clean_tree_results.failing_tests())) <= 5: >+ return False > >- tests_that_only_failed_second = second_failing_results_set.difference(first_failing_results_set) >- self._report_flaky_tests(tests_that_only_failed_second, second_results_archive) >+ self.failure_status_id = first_failure_status_id > >- tests_that_consistently_failed = first_failing_results_set.intersection(second_failing_results_set) >- if tests_that_consistently_failed: >- if self._should_defer_patch_or_throw(tests_that_consistently_failed, first_results_archive, first_script_error, first_failure_status_id): >- return False # Defer patch >+ return self.report_failure(first_results_archive, first_results, first_script_error) > >- # At this point we know that at least one test flaked, but no consistent failures >- # were introduced. This is a bit of a grey-zone. >- return False # Defer patch >+ if second_results.did_exceed_test_failure_limit(): >+ self._should_defer_patch_or_throw(first_results.failing_test_results(), first_results_archive, first_script_error, first_failure_status_id) >+ return False >+ >+ if first_results.did_exceed_test_failure_limit(): >+ self._should_defer_patch_or_throw(second_results.failing_test_results(), second_results_archive, second_script_error, second_failure_status_id) >+ return False >+ >+ if self._results_failed_different_tests(first_results, second_results): >+ first_failing_results_set = frozenset(first_results.failing_test_results()) >+ second_failing_results_set = frozenset(second_results.failing_test_results()) >+ >+ tests_that_only_failed_first = first_failing_results_set.difference(second_failing_results_set) >+ self._report_flaky_tests(tests_that_only_failed_first, first_results_archive) > >- if self._should_defer_patch_or_throw(first_results.failing_test_results(), first_results_archive, first_script_error, first_failure_status_id): >- return False # Defer patch >+ tests_that_only_failed_second = second_failing_results_set.difference(first_failing_results_set) >+ self._report_flaky_tests(tests_that_only_failed_second, second_results_archive) > >- # At this point, we know that the first and second runs had the exact same failures, >- # and that those failures are all present on the clean tree, so we can say with certainty >- # that the patch is good. >- return True >+ tests_that_consistently_failed = first_failing_results_set.intersection(second_failing_results_set) >+ if tests_that_consistently_failed: >+ if self._should_defer_patch_or_throw(tests_that_consistently_failed, first_results_archive, first_script_error, first_failure_status_id): >+ return False # Defer patch >+ >+ # At this point we know that at least one test flaked, but no consistent failures >+ # were introduced. This is a bit of a grey-zone. >+ return False # Defer patch >+ >+ if self._should_defer_patch_or_throw(first_results.failing_test_results(), first_results_archive, first_script_error, first_failure_status_id): >+ return False # Defer patch >+ >+ # At this point, we know that the first and second runs had the exact same failures, >+ # and that those failures are all present on the clean tree, so we can say with certainty >+ # that the patch is good. >+ return True > > def results_archive_from_patch_test_run(self, patch): > assert(self._patch.id() == patch.id()) # PatchAnalysisTask is not currently re-useable. >diff --git a/Tools/Scripts/webkitpy/tool/commands/download.py b/Tools/Scripts/webkitpy/tool/commands/download.py >index e5169aac85f3186e84be9d5024592bd7fe6d5dd5..14b55bc3f7c9c10b5df60fc1af87d7ca39e3bc94 100644 >--- a/Tools/Scripts/webkitpy/tool/commands/download.py >+++ b/Tools/Scripts/webkitpy/tool/commands/download.py >@@ -1,5 +1,5 @@ > # Copyright (c) 2009, 2011 Google Inc. All rights reserved. >-# Copyright (c) 2009 Apple Inc. All rights reserved. >+# Copyright (c) 2009, 2017 Apple Inc. All rights reserved. > # > # Redistribution and use in source and binary forms, with or without > # modification, are permitted provided that the following conditions are >@@ -87,6 +87,14 @@ class BuildAndTest(AbstractSequencedCommand): > ] > > >+class CheckPatchRelevance(AbstractSequencedCommand): >+ name = "check-patch-relevance" >+ help_text = "Check if this patch needs to be tested" >+ steps = [ >+ steps.CheckPatchRelevance, >+ ] >+ >+ > class Land(AbstractSequencedCommand): > name = "land" > help_text = "Land the current working directory diff and updates the associated bug if any" >diff --git a/Tools/Scripts/webkitpy/tool/commands/earlywarningsystem.py b/Tools/Scripts/webkitpy/tool/commands/earlywarningsystem.py >index 9bc041860ce67454454e4b251438a9dbe5264513..6aa2f9fc298221796c871eab2cc13e020f424021 100644 >--- a/Tools/Scripts/webkitpy/tool/commands/earlywarningsystem.py >+++ b/Tools/Scripts/webkitpy/tool/commands/earlywarningsystem.py >@@ -1,4 +1,5 @@ > # Copyright (c) 2009 Google Inc. All rights reserved. >+# Copyright (c) 2017 Apple Inc. All rights reserved. > # > # Redistribution and use in source and binary forms, with or without > # modification, are permitted provided that the following conditions are >@@ -28,6 +29,7 @@ > > import json > import logging >+import os > from optparse import make_option > > from webkitpy.common.config.committers import CommitterList >@@ -36,7 +38,8 @@ from webkitpy.common.system.filesystem import FileSystem > from webkitpy.common.system.executive import ScriptError > from webkitpy.tool.bot.earlywarningsystemtask import EarlyWarningSystemTask, EarlyWarningSystemTaskDelegate > from webkitpy.tool.bot.layouttestresultsreader import LayoutTestResultsReader >-from webkitpy.tool.bot.patchanalysistask import UnableToApplyPatch, PatchIsNotValid >+from webkitpy.tool.bot.jsctestresultsreader import JSCTestResultsReader >+from webkitpy.tool.bot.patchanalysistask import UnableToApplyPatch, PatchIsNotValid, PatchIsNotApplicable > from webkitpy.tool.bot.queueengine import QueueEngine > from webkitpy.tool.commands.queues import AbstractReviewQueue > >@@ -53,7 +56,12 @@ class AbstractEarlyWarningSystem(AbstractReviewQueue, EarlyWarningSystemTaskDele > > def begin_work_queue(self): > AbstractReviewQueue.begin_work_queue(self) >- self._layout_test_results_reader = LayoutTestResultsReader(self._tool, self._port.results_directory(), self._log_directory()) >+ >+ if self.group() == "jsc": >+ dir = os.path.dirname(os.path.dirname(os.path.dirname(self._port.results_directory()))) >+ self._test_results_reader = JSCTestResultsReader(self._tool, dir) >+ else: >+ self._test_results_reader = LayoutTestResultsReader(self._tool, self._port.results_directory(), self._log_directory()) > > def _failing_tests_message(self, task, patch): > results = task.results_from_patch_test_run(patch) >@@ -93,6 +101,9 @@ class AbstractEarlyWarningSystem(AbstractReviewQueue, EarlyWarningSystemTaskDele > except UnableToApplyPatch, e: > self._did_error(patch, "%s unable to apply patch." % self.name) > return False >+ except PatchIsNotApplicable, e: >+ self._did_error(patch, "%s patch not applicable." % self.name) >+ return False > except ScriptError, e: > self._post_reject_message_on_bug(self._tool, patch, task.failure_status_id, self._failing_tests_message(task, patch)) > results_archive = task.results_archive_from_patch_test_run(patch) >@@ -117,14 +128,17 @@ class AbstractEarlyWarningSystem(AbstractReviewQueue, EarlyWarningSystemTaskDele > return self._update_status(message, patch=patch, results_file=failure_log) > > def test_results(self): >- return self._layout_test_results_reader.results() >+ return self._test_results_reader.results() > > def archive_last_test_results(self, patch): >- return self._layout_test_results_reader.archive(patch) >+ return self._test_results_reader.archive(patch) > > def build_style(self): > return self._build_style > >+ def group(self): >+ return self._group >+ > def refetch_patch(self, patch): > return self._tool.bugs.fetch_attachment(patch.id()) > >@@ -156,5 +170,6 @@ class AbstractEarlyWarningSystem(AbstractReviewQueue, EarlyWarningSystemTaskDele > '_build_style': config.get('style', "release"), > 'watchers': config.get('watchers', []), > 'run_tests': config.get('runTests', cls.run_tests), >+ '_group': config.get('group', None), > })) > return classes >diff --git a/Tools/Scripts/webkitpy/tool/commands/earlywarningsystem_unittest.py b/Tools/Scripts/webkitpy/tool/commands/earlywarningsystem_unittest.py >index 08887813ce2e93a8ec9da37636ecbde9eff3c9e7..5d78a8170bfab6d32764e5dd22cf5e7014db6328 100644 >--- a/Tools/Scripts/webkitpy/tool/commands/earlywarningsystem_unittest.py >+++ b/Tools/Scripts/webkitpy/tool/commands/earlywarningsystem_unittest.py >@@ -1,4 +1,5 @@ > # Copyright (C) 2009 Google Inc. All rights reserved. >+# Copyright (C) 2017 Apple Inc. All rights reserved. > # > # Redistribution and use in source and binary forms, with or without > # modification, are permitted provided that the following conditions are >@@ -29,6 +30,7 @@ > from webkitpy.thirdparty.mock import Mock > from webkitpy.common.host import Host > from webkitpy.common.host_mock import MockHost >+from webkitpy.common.net.jsctestresults import JSCTestResults > from webkitpy.common.net.layouttestresults import LayoutTestResults > from webkitpy.common.system.outputcapture import OutputCapture > from webkitpy.layout_tests.models import test_results >@@ -47,6 +49,7 @@ class AbstractEarlyWarningSystemTest(QueuesTest): > class TestEWS(AbstractEarlyWarningSystem): > port_name = "win" # Needs to be a port which port/factory understands. > _build_style = None >+ _group = None > > ews = TestEWS() > ews.bind_to_tool(MockTool()) >@@ -60,6 +63,23 @@ class AbstractEarlyWarningSystemTest(QueuesTest): > patch = ews._tool.bugs.fetch_attachment(10000) > self.assertMultiLineEqual(ews._failing_tests_message(task, patch), "New failing tests:\nfoo.html\nbar.html") > >+ def test_failing_jsc_tests_message(self): >+ # Needed to define port_name, used in AbstractEarlyWarningSystem.__init__ >+ class TestEWS(AbstractEarlyWarningSystem): >+ port_name = "win" # Needs to be a port which port/factory understands. >+ _build_style = None >+ _group = "jsc" >+ >+ ews = TestEWS() >+ ews.bind_to_tool(MockTool()) >+ ews.host = MockHost() >+ ews._options = MockOptions(port=None, confirm=False) >+ OutputCapture().assert_outputs(self, ews.begin_work_queue, expected_logs=self._default_begin_work_queue_logs(ews.name)) >+ task = Mock() >+ task.results_from_patch_test_run = lambda a: JSCTestResults(False, ["es6.yaml/es6/typed_arrays_Int8Array.js.default", "es6.yaml/es6/typed_arrays_Uint8Array.js.default"]) >+ patch = ews._tool.bugs.fetch_attachment(10000) >+ self.assertMultiLineEqual(ews._failing_tests_message(task, patch), "New failing tests:\nes6.yaml/es6/typed_arrays_Int8Array.js.default\nes6.yaml/es6/typed_arrays_Uint8Array.js.default\napiTests") >+ > > class EarlyWarningSystemTest(QueuesTest): > def _default_expected_logs(self, ews): >@@ -68,9 +88,10 @@ class EarlyWarningSystemTest(QueuesTest): > "port": ews.port_name, > "architecture": " --architecture=%s" % ews.architecture if ews.architecture else "", > "build_style": ews.build_style(), >+ "group": ews.group(), > } > if ews.run_tests: >- run_tests_line = "Running: webkit-patch --status-host=example.com build-and-test --no-clean --no-update --test --non-interactive --build-style=%(build_style)s --port=%(port)s%(architecture)s\n" % string_replacements >+ run_tests_line = "Running: webkit-patch --status-host=example.com build-and-test --no-clean --no-update --test --non-interactive --build-style=%(build_style)s --group=%(group)s --port=%(port)s%(architecture)s\n" % string_replacements > else: > run_tests_line = "" > string_replacements['run_tests_line'] = run_tests_line >@@ -81,7 +102,8 @@ class EarlyWarningSystemTest(QueuesTest): > Running: webkit-patch --status-host=example.com clean --port=%(port)s%(architecture)s > Running: webkit-patch --status-host=example.com update --port=%(port)s%(architecture)s > Running: webkit-patch --status-host=example.com apply-attachment --no-update --non-interactive 10000 --port=%(port)s%(architecture)s >-Running: webkit-patch --status-host=example.com build --no-clean --no-update --build-style=%(build_style)s --port=%(port)s%(architecture)s >+Running: webkit-patch --status-host=example.com check-patch-relevance --group=%(group)s --port=%(port)s%(architecture)s >+Running: webkit-patch --status-host=example.com build --no-clean --no-update --build-style=%(build_style)s --group=%(group)s --port=%(port)s%(architecture)s > %(run_tests_line)sMOCK: update_status: %(name)s Pass > MOCK: release_work_item: %(name)s 10000 > """ % string_replacements, >diff --git a/Tools/Scripts/webkitpy/tool/steps/__init__.py b/Tools/Scripts/webkitpy/tool/steps/__init__.py >index 356acab03ad8274ce9bda6858864ff19b64f4a0a..6c9c09ac52a386b9f0cf363b9c03bbaa4c003861 100644 >--- a/Tools/Scripts/webkitpy/tool/steps/__init__.py >+++ b/Tools/Scripts/webkitpy/tool/steps/__init__.py >@@ -1,4 +1,5 @@ > # Copyright (C) 2010 Google Inc. All rights reserved. >+# Copyright (C) 2017 Apple Inc. All rights reserved. > # > # Redistribution and use in source and binary forms, with or without > # modification, are permitted provided that the following conditions are >@@ -33,6 +34,7 @@ from webkitpy.tool.steps.applypatchwithlocalcommit import ApplyPatchWithLocalCom > from webkitpy.tool.steps.applywatchlist import ApplyWatchList > from webkitpy.tool.steps.attachtobug import AttachToBug > from webkitpy.tool.steps.build import Build >+from webkitpy.tool.steps.checkpatchrelevance import CheckPatchRelevance > from webkitpy.tool.steps.checkstyle import CheckStyle > from webkitpy.tool.steps.cleanworkingdirectory import CleanWorkingDirectory > from webkitpy.tool.steps.closebug import CloseBug >diff --git a/Tools/Scripts/webkitpy/tool/steps/build.py b/Tools/Scripts/webkitpy/tool/steps/build.py >index f74ffce70941c61e596792e4c60dcd7f8b65e9ec..3c6cf0b1de018c132b7f86da2cb907e2fda067ae 100644 >--- a/Tools/Scripts/webkitpy/tool/steps/build.py >+++ b/Tools/Scripts/webkitpy/tool/steps/build.py >@@ -1,4 +1,5 @@ > # Copyright (C) 2010 Google Inc. All rights reserved. >+# Copyright (C) 2017 Apple Inc. All rights reserved. > # > # Redistribution and use in source and binary forms, with or without > # modification, are permitted provided that the following conditions are >@@ -41,28 +42,37 @@ class Build(AbstractStep): > Options.build, > Options.quiet, > Options.build_style, >+ Options.group, > ] > >- def build(self, build_style): >+ def build(self, build_style, group): > environment = self._tool.copy_current_environment() > environment.disable_gcc_smartquotes() > environment.disable_jhbuild_VT100_output() > env = environment.to_dictionary() > >- build_webkit_command = self._tool.deprecated_port().build_webkit_command(build_style=build_style) >+ if group == "jsc": >+ build_command = self._tool.deprecated_port().build_jsc_command(build_style=build_style) >+ else: >+ build_command = self._tool.deprecated_port().build_webkit_command(build_style=build_style) > > if self._options.architecture: >- build_webkit_command += ['ARCHS=%s' % self._options.architecture] >+ build_command += ['ARCHS=%s' % self._options.architecture] > >- self._tool.executive.run_and_throw_if_fail(build_webkit_command, self._options.quiet, >+ self._tool.executive.run_and_throw_if_fail(build_command, self._options.quiet, > cwd=self._tool.scm().checkout_root, env=env) > > def run(self, state): > if not self._options.build: > return > _log.info("Building WebKit") >+ >+ group = None >+ if hasattr(self._options, 'group'): >+ group = self._options.group >+ > if self._options.build_style == "both": >- self.build("debug") >- self.build("release") >+ self.build("debug", group) >+ self.build("release", group) > else: >- self.build(self._options.build_style) >+ self.build(self._options.build_style, group) >diff --git a/Tools/Scripts/webkitpy/tool/steps/checkpatchrelevance.py b/Tools/Scripts/webkitpy/tool/steps/checkpatchrelevance.py >new file mode 100644 >index 0000000000000000000000000000000000000000..82f745e34db9d69b2445348844685966ea663252 >--- /dev/null >+++ b/Tools/Scripts/webkitpy/tool/steps/checkpatchrelevance.py >@@ -0,0 +1,74 @@ >+# Copyright (C) 2017 Apple Inc. All rights reserved. >+# >+# Redistribution and use in source and binary forms, with or without >+# modification, are permitted provided that the following conditions >+# are met: >+# 1. Redistributions of source code must retain the above copyright >+# notice, this list of conditions and the following disclaimer. >+# 2. Redistributions in binary form must reproduce the above copyright >+# notice, this list of conditions and the following disclaimer in the >+# documentation and/or other materials provided with the distribution. >+# >+# THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS'' AND >+# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED >+# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE >+# DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS BE LIABLE FOR >+# ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL >+# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR >+# SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER >+# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, >+# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE >+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. >+ >+import logging >+import re >+ >+from webkitpy.common.system.executive import ScriptError >+from webkitpy.tool.steps.abstractstep import AbstractStep >+from webkitpy.tool.steps.options import Options >+ >+_log = logging.getLogger(__name__) >+ >+ >+class CheckPatchRelevance(AbstractStep): >+ @classmethod >+ def options(cls): >+ return AbstractStep.options() + [ >+ Options.group, >+ ] >+ >+ jsc_paths = [ >+ "JSTests/", >+ "Source/JavaScriptCore/", >+ "Source/WTF/" >+ "Source/bmalloc/", >+ ] >+ >+ def _does_contain_change_in_paths(self, files, patterns): >+ for item in files: >+ for pattern in patterns: >+ if re.search(pattern, item, re.IGNORECASE): >+ return True >+ >+ return False >+ >+ def run(self, state): >+ _log.info("Checking relevance of patch") >+ >+ group = None >+ if hasattr(self._options, 'group'): >+ group = self._options.group >+ >+ change_list = self._tool.scm().changed_files() >+ >+ if group == "jsc": >+ relevant = self._does_contain_change_in_paths(change_list, self.jsc_paths) >+ else: >+ relevant = True >+ >+ if relevant is True: >+ _log.info("This patch has relevant changes.") >+ return True >+ else: >+ _log.info("This patch does not have relevant changes.") >+ raise ScriptError(message="This patch does not have relevant changes.") >diff --git a/Tools/Scripts/webkitpy/tool/steps/options.py b/Tools/Scripts/webkitpy/tool/steps/options.py >index 3ef47d008a59e050d4ca0c4877ddd1a60731a409..f5e0def32717e306e63169841110b2898b72a1f1 100644 >--- a/Tools/Scripts/webkitpy/tool/steps/options.py >+++ b/Tools/Scripts/webkitpy/tool/steps/options.py >@@ -1,4 +1,5 @@ > # Copyright (C) 2010 Google Inc. All rights reserved. >+# Copyright (C) 2017 Apple Inc. All rights reserved. > # > # Redistribution and use in source and binary forms, with or without > # modification, are permitted provided that the following conditions are >@@ -46,6 +47,7 @@ class Options(object): > email = make_option("--email", action="store", type="string", dest="email", help="Email address to use in ChangeLogs.") > force_clean = make_option("--force-clean", action="store_true", dest="force_clean", default=False, help="Clean working directory before applying patches (removes local changes and commits)") > git_commit = make_option("-g", "--git-commit", action="store", dest="git_commit", help="Operate on a local commit. If a range, the commits are squashed into one. <ref>.... includes the working copy changes. UPSTREAM can be used for the upstream/tracking branch.") >+ group = make_option("--group", action="store", dest="group", default=None, help="") > local_commit = make_option("--local-commit", action="store_true", dest="local_commit", default=False, help="Make a local commit for each applied patch") > non_interactive = make_option("--non-interactive", action="store_true", dest="non_interactive", default=False, help="Never prompt the user, fail as fast as possible.") > obsolete_patches = make_option("--no-obsolete", action="store_false", dest="obsolete_patches", default=True, help="Do not obsolete old patches before posting this one.") >diff --git a/Tools/Scripts/webkitpy/tool/steps/runtests.py b/Tools/Scripts/webkitpy/tool/steps/runtests.py >index 75d72bc9ad2f4e36d45fdac90f1819481cf0288b..5a7f2a98a3fbb24ecb03f61c6b60f65042d971a4 100644 >--- a/Tools/Scripts/webkitpy/tool/steps/runtests.py >+++ b/Tools/Scripts/webkitpy/tool/steps/runtests.py >@@ -1,4 +1,5 @@ > # Copyright (C) 2010 Google Inc. All rights reserved. >+# Copyright (C) 2017 Apple Inc. All rights reserved. > # > # Redistribution and use in source and binary forms, with or without > # modification, are permitted provided that the following conditions are >@@ -53,6 +54,7 @@ class RunTests(AbstractStep): > Options.iterate_on_new_tests, > Options.non_interactive, > Options.quiet, >+ Options.group, > ] > > def run(self, state): >@@ -63,6 +65,10 @@ class RunTests(AbstractStep): > if not self._options.test: > return > >+ if hasattr(self._options, "group") and self._options.group == "jsc": >+ self._run_javascriptcore_tests() >+ return >+ > python_unittests_command = self._tool.deprecated_port().run_python_unittests_command() > if python_unittests_command: > _log.info("Running Python unit tests") >@@ -143,3 +149,7 @@ class RunTests(AbstractStep): > args.append("--iterations=%d" % iterate_on_new_tests) > > self._tool.executive.run_and_throw_if_fail(args, cwd=self._tool.scm().checkout_root) >+ >+ def _run_javascriptcore_tests(self): >+ args = self._tool.deprecated_port().run_javascriptcore_tests_command(self._options.build_style) >+ self._tool.executive.run_and_throw_if_fail(args, cwd=self._tool.scm().checkout_root) >diff --git a/Tools/Scripts/webkitpy/tool/steps/steps_unittest.py b/Tools/Scripts/webkitpy/tool/steps/steps_unittest.py >index 7d94a13b3463d98a44f949f814cc7121c068ab5d..c3300c85944cde21ad9d0b3e56ded88013eb6ade 100644 >--- a/Tools/Scripts/webkitpy/tool/steps/steps_unittest.py >+++ b/Tools/Scripts/webkitpy/tool/steps/steps_unittest.py >@@ -1,4 +1,5 @@ > # Copyright (C) 2010 Google Inc. All rights reserved. >+# Copyright (C) 2017 Apple Inc. All rights reserved. > # > # Redistribution and use in source and binary forms, with or without > # modification, are permitted provided that the following conditions are >@@ -28,6 +29,7 @@ > > import unittest > >+from webkitpy.common.system.executive import ScriptError > from webkitpy.common.system.outputcapture import OutputCapture > from webkitpy.common.config.ports import DeprecatedPort > from webkitpy.tool.mocktool import MockOptions, MockTool >@@ -111,7 +113,7 @@ MOCK run_and_throw_if_fail: ['Tools/Scripts/test-webkitpy'], cwd=/mock-checkout > Running Perl unit tests > MOCK run_and_throw_if_fail: ['Tools/Scripts/test-webkitperl'], cwd=/mock-checkout > Running JavaScriptCore tests >-MOCK run_and_throw_if_fail: ['Tools/Scripts/run-javascriptcore-tests'], cwd=/mock-checkout >+MOCK run_and_throw_if_fail: ['Tools/Scripts/run-javascriptcore-tests', '--no-fail-fast', '--json-output=jsc_test_results.json'], cwd=/mock-checkout > Running bindings generation tests > MOCK run_and_throw_if_fail: ['Tools/Scripts/run-bindings-tests'], cwd=/mock-checkout > Running run-webkit-tests >@@ -133,10 +135,111 @@ MOCK run_and_throw_if_fail: ['Tools/Scripts/test-webkitpy'], cwd=/mock-checkout > Running Perl unit tests > MOCK run_and_throw_if_fail: ['Tools/Scripts/test-webkitperl'], cwd=/mock-checkout > Running JavaScriptCore tests >-MOCK run_and_throw_if_fail: ['Tools/Scripts/run-javascriptcore-tests'], cwd=/mock-checkout >+MOCK run_and_throw_if_fail: ['Tools/Scripts/run-javascriptcore-tests', '--no-fail-fast', '--json-output=jsc_test_results.json'], cwd=/mock-checkout > Running bindings generation tests > MOCK run_and_throw_if_fail: ['Tools/Scripts/run-bindings-tests'], cwd=/mock-checkout > Running run-webkit-tests > MOCK run_and_throw_if_fail: ['Tools/Scripts/run-webkit-tests', '--debug', '--quiet'], cwd=/mock-checkout > """ > OutputCapture().assert_outputs(self, step.run, [{}], expected_logs=expected_logs) >+ >+ def test_runtests_jsc(self): >+ mock_options = self._step_options() >+ mock_options.non_interactive = False >+ mock_options.build_style = "release" >+ mock_options.group = "jsc" >+ step = steps.RunTests(MockTool(log_executive=True), mock_options) >+ tool = MockTool(log_executive=True) >+ # FIXME: We shouldn't use a real port-object here, but there is too much to mock at the moment. >+ tool._deprecated_port = DeprecatedPort() >+ step = steps.RunTests(tool, mock_options) >+ expected_logs = """MOCK run_and_throw_if_fail: ['Tools/Scripts/run-javascriptcore-tests', '--no-fail-fast', '--json-output=jsc_test_results.json'], cwd=/mock-checkout >+""" >+ OutputCapture().assert_outputs(self, step.run, [{}], expected_logs=expected_logs) >+ >+ def test_runtests_jsc_debug(self): >+ mock_options = self._step_options() >+ mock_options.non_interactive = False >+ mock_options.build_style = "debug" >+ mock_options.group = "jsc" >+ tool = MockTool(log_executive=True) >+ # FIXME: We shouldn't use a real port-object here, but there is too much to mock at the moment. >+ tool._deprecated_port = DeprecatedPort() >+ step = steps.RunTests(tool, mock_options) >+ expected_logs = """MOCK run_and_throw_if_fail: ['Tools/Scripts/run-javascriptcore-tests', '--no-fail-fast', '--json-output=jsc_test_results.json', '--debug'], cwd=/mock-checkout >+""" >+ OutputCapture().assert_outputs(self, step.run, [{}], expected_logs=expected_logs) >+ >+ def test_build_jsc_debug(self): >+ mock_options = self._step_options() >+ mock_options.non_interactive = False >+ mock_options.build_style = "debug" >+ mock_options.build = True >+ mock_options.architecture = True >+ mock_options.group = "jsc" >+ tool = MockTool(log_executive=True) >+ # FIXME: We shouldn't use a real port-object here, but there is too much to mock at the moment. >+ tool._deprecated_port = DeprecatedPort() >+ step = steps.Build(tool, mock_options) >+ expected_logs = """Building WebKit >+MOCK run_and_throw_if_fail: ['Tools/Scripts/build-jsc', '--debug', 'ARCHS=True'], cwd=/mock-checkout, env={'LC_ALL': 'C', 'TERM': 'none', 'MOCK_ENVIRON_COPY': '1'} >+""" >+ OutputCapture().assert_outputs(self, step.run, [{}], expected_logs=expected_logs) >+ >+ def test_build_jsc(self): >+ mock_options = self._step_options() >+ mock_options.non_interactive = False >+ mock_options.build_style = "release" >+ mock_options.build = True >+ mock_options.architecture = True >+ mock_options.group = "jsc" >+ tool = MockTool(log_executive=True) >+ # FIXME: We shouldn't use a real port-object here, but there is too much to mock at the moment. >+ tool._deprecated_port = DeprecatedPort() >+ step = steps.Build(tool, mock_options) >+ expected_logs = """Building WebKit >+MOCK run_and_throw_if_fail: ['Tools/Scripts/build-jsc', '--release', 'ARCHS=True'], cwd=/mock-checkout, env={'LC_ALL': 'C', 'TERM': 'none', 'MOCK_ENVIRON_COPY': '1'} >+""" >+ OutputCapture().assert_outputs(self, step.run, [{}], expected_logs=expected_logs) >+ >+ def test_patch_relevant(self): >+ self.maxDiff = None >+ mock_options = self._step_options() >+ tool = MockTool(log_executive=True) >+ tool.scm()._mockChangedFiles = ["JSTests/MockFile1", "ChangeLog"] >+ # FIXME: We shouldn't use a real port-object here, but there is too much to mock at the moment. >+ tool._deprecated_port = DeprecatedPort() >+ step = steps.CheckPatchRelevance(tool, mock_options) >+ expected_logs = """Checking relevance of patch >+This patch has relevant changes. >+""" >+ OutputCapture().assert_outputs(self, step.run, [{}], expected_logs=expected_logs) >+ >+ def test_patch_relevant_jsc(self): >+ self.maxDiff = None >+ mock_options = self._step_options() >+ mock_options.group = "jsc" >+ tool = MockTool(log_executive=True) >+ tool.scm()._mockChangedFiles = ["JSTests/MockFile1", "ChangeLog"] >+ # FIXME: We shouldn't use a real port-object here, but there is too much to mock at the moment. >+ tool._deprecated_port = DeprecatedPort() >+ step = steps.CheckPatchRelevance(tool, mock_options) >+ expected_logs = """Checking relevance of patch >+This patch has relevant changes. >+""" >+ OutputCapture().assert_outputs(self, step.run, [{}], expected_logs=expected_logs) >+ >+ def test_patch_not_relevant_jsc(self): >+ self.maxDiff = None >+ mock_options = self._step_options() >+ mock_options.group = "jsc" >+ tool = MockTool(log_executive=True) >+ tool.scm()._mockChangedFiles = ["Tools/ChangeLog", "Tools/Scripts/webkitpy/tool/steps/steps_unittest.py"] >+ # FIXME: We shouldn't use a real port-object here, but there is too much to mock at the moment. >+ tool._deprecated_port = DeprecatedPort() >+ step = steps.CheckPatchRelevance(tool, mock_options) >+ expected_logs = """Checking relevance of patch >+This patch does not have relevant changes. >+""" >+ with self.assertRaises(ScriptError): >+ OutputCapture().assert_outputs(self, step.run, [{}], expected_logs=expected_logs)
<b>You cannot view the attachment while viewing its details because your browser does not support IFRAMEs. <a href="attachment.cgi?id=300474">View the attachment on a separate page</a>.</b>
View Attachment As Diff
View Attachment As Raw
Actions:
View
|
Formatted Diff
|
Diff
Attachments on
bug 162458
:
297210
|
297239
|
297551
|
298160
|
298714
|
298739
|
298802
|
298807
|
300176
|
300294
|
300461
|
300474
|
300784
|
300838
|
301020
|
301069
|
301117
|
301137
|
301437
|
301572
|
301667
|
301848
|
301849
|
301972
|
302000