Attachment #300474 for bug #162458

View Details Raw Unified Return to Bug 162458

| Differences between



2017-01-30  Srinivasan Vijayaraghavan  <svijayaraghavan@apple.com>

        EWS should run JavaScriptCore tests
        https://bugs.webkit.org/show_bug.cgi?id=162458

        Reviewed by NOBODY (OOPS!).

        * Scripts/webkitpy/common/checkout/scm/scm_mock.py:
        (MockSCM.__init__): Add _mockChangedFiles variable.
        (MockSCM.changed_files): Change to use _mockChangedFiles variables.
        * Scripts/webkitpy/common/config/ews.json: Add config info for JSC EWS.
        * Scripts/webkitpy/common/config/ports.py: Add support for JSC EWS in Mac Port.
        (DeprecatedPort.build_jsc_command): Added - command to build JSC only (quicker than building all of WebKit).
        (DeprecatedPort.run_javascriptcore_tests_command): Allow JSC EWS to only run JSC tests.
        (MacPort.run_webkit_tests_command): Check for JSC.
        * Scripts/webkitpy/common/config/ports_mock.py:
        (MockPort.run_javascriptcore_tests_command): Add build_style argument.
        * Scripts/webkitpy/common/config/ports_unittest.py:
        (DeprecatedPortTest.test_mac_port): Add unit tests for build-jsc shell command.
        * Scripts/webkitpy/common/net/jsctestresults.py: Added.
        (JSCTestResults): Added.
        (JSCTestResults.intersection): Return a JSCTestResults object with failures common to both input objects.
        (JSCTestResults.results_from_string): Use json library to parse results.
        (JSCTestResults.__init__): Initialize members.
        (JSCTestResults.equals): This enables unit testing.
        (JSCTestResults.is_subset): Checks if one set of failures is a subset of another.
        (JSCTestResults.stress_test_failures): Getter.
        (JSCTestResults.all_api_tests_passed): Getter.
        (JSCTestResults.all_passed): Getter.
        (JSCTestResults.failing_test_results): Getter. Some of these exist for interopability reasons.
        (JSCTestResults.test_results): Getter.
        (JSCTestResults.failing_tests): Getter.
        (JSCTestResults.did_exceed_test_failure_limit): Getter.
        * Scripts/webkitpy/common/net/jsctestresults_unittest.py: Added.
        (JSCTestResultsTest): Class to test JSCTestResults.
        (JSCTestResultsTest.test_results_from_string): Tests parsing.
        (JSCTestResultsTest.test_intersection_api_tests): Unit test for intersection() class method.
        (JSCTestResultsTest.test_intersection_stress_tests): Unit test for intersection() class method.
        (JSCTestResultsTest.test_intersection_general_case): Unit test for intersection() class method.
        * Scripts/webkitpy/tool/bot/earlywarningsystemtask.py:
        (EarlyWarningSystemTask.run): Add check_patch_relevance step.
        * Scripts/webkitpy/tool/bot/jscews_unittest.py: Added.
        (MockPatchAnalysisTask): Mocked-out version of PatchAnalysisTask that doesn't run commands.
        (MockPatchAnalysisTask.__init__): Sets attributes.
        (MockPatchAnalysisTask._test): Override to not run command.
        (MockPatchAnalysisTask._build_and_test_without_patch): Override to not run command.
        (MockPatchAnalysisTask.validate): Assume mocked patch is valid for purposes of testing retry logic.
        (MockPatchAnalysisTask.test_run_count): Specific to the mocked version, to test retry logic.
        (MockJSCEarlyWarningSystem): Mocked-out version of AbstarctEarlyWarningSystem so we can provide test results.
        (MockJSCEarlyWarningSystem.__init__): Sets attributes, also sets group to jsc.
        (MockJSCEarlyWarningSystem.test_results): Returns test results provided by us, instead of using a JSON reader.
        (JSCEarlyWarningSystemTest): Class to test retry logic in below situations.
        (JSCEarlyWarningSystemTest._create_task): Helper function to abstract out common code.
        (JSCEarlyWarningSystemTest._results_indicate_all_passed): False if input is None or has failures, else True.
        (JSCEarlyWarningSystemTest.test_success_case): Clean patch on clean tree.
        (JSCEarlyWarningSystemTest.test_test_failure): Failed patch on clean tree.
        (JSCEarlyWarningSystemTest.test_fix): Patch that fixes all tree redness.
        (JSCEarlyWarningSystemTest.test_ineffective_patch): Patch that has same failures as tree.
        (JSCEarlyWarningSystemTest.test_partially_effective_patch): Patch fixes some failures but adds no new failures.
        (JSCEarlyWarningSystemTest.test_different_test_failures_in_patch_and_tree): Patch has some failures not in tree.
        (JSCEarlyWarningSystemTest.test_first_results_could_not_be_read): Patch results not readable.
        (JSCEarlyWarningSystemTest.test_second_results_could_not_be_read): Patch results not readable on second run.
        (JSCEarlyWarningSystemTest.test_clean_results_could_not_be_read): Results from clean tree not readable.
        (JSCEarlyWarningSystemTest.test_flaky_results_on_clean_tree_pass): Patch has one flake and no failures.
        (JSCEarlyWarningSystemTest.test_flaky_results_on_clean_tree_pass_v2): Patch has one flake and no failures.
        (JSCEarlyWarningSystemTest.test_flaky_results_on_clean_tree_failure): Patch has flakes and failed tests.
        (JSCEarlyWarningSystemTest.test_flaky_results_on_red_tree_pass): Patch has same failures as tree, plus a flake.
        * Scripts/webkitpy/tool/bot/jsctestresultsreader.py: Added.
        (JSCTestResultsReader): Reads results file.
        (JSCTestResultsReader.__init__): Sets attributes.
        (JSCTestResultsReader._read_file_contents): Reads file.
        (JSCTestResultsReader._create_jsc_test_results): Creates a JSCTestResults object.
        (JSCTestResultsReader.results): Invokes _create_jsc_test_results to return a JSCTestResults object.
        * Scripts/webkitpy/tool/bot/patchanalysistask.py:
        (PatchIsNotApplicable): Exception for when patch doesn't have relevant changes.
        (PatchAnalysisTask._check_patch_relevance): Added.
        (PatchAnalysisTask._build): Check for JSC.
        (PatchAnalysisTask._build_without_patch): Check for JSC.
        (PatchAnalysisTask._test): Check for JSC.
        (PatchAnalysisTask._build_and_test_without_patch): Check for JSC.
        (PatchAnalysisTask._test_patch): Add retry logic for JSC.
        * Scripts/webkitpy/tool/commands/download.py:
        (CheckPatchRelevance): Add check-patch-relevance command.
        * Scripts/webkitpy/tool/commands/earlywarningsystem.py:
        (AbstractEarlyWarningSystem.begin_work_queue): Use JSCTestResultsReader not LayoutTestResultsReader in JSC EWS.
        (AbstractEarlyWarningSystem.review_patch): Handle PatchIsApplicable
        (AbstractEarlyWarningSystem.test_results): _layout_test_results_reader -> _test_results_reader.
        (AbstractEarlyWarningSystem.archive_last_test_results): _layout_test_results_reader -> _test_results_reader.
        (AbstractEarlyWarningSystem.group): This attribute determines the type of EWS (eg. JSC).
        (AbstractEarlyWarningSystem.load_ews_classes): Add _group.
        * Scripts/webkitpy/tool/commands/earlywarningsystem_unittest.py:
        (AbstractEarlyWarningSystemTest.test_failing_tests_message.TestEWS): Add _group.
        (AbstractEarlyWarningSystemTest.test_failing_jsc_tests_message.TestEWS): Added test for jsc failures message.
        (AbstractEarlyWarningSystemTest): Add _group variable.
        (EarlyWarningSystemTest._default_expected_logs): Add check-patch-relevance step and --group option.
        * Scripts/webkitpy/tool/steps/__init__.py: Add CheckPatchRelevance import.
        * Scripts/webkitpy/tool/steps/build.py:
        (Build.options): Check for JSC.
        (Build.build): Check for JSC.
        (Build.run): Check for JSC.
        * Scripts/webkitpy/tool/steps/checkpatchrelevance.py: Added.
        (CheckPatchRelevance): Added.
        (CheckPatchRelevance._does_contain_change_in_paths): Abstract function to perform regex matching.
        (CheckPatchRelevance.run): Check if changed files in patch belong in certain folders.
        * Scripts/webkitpy/tool/steps/options.py:
        (Options): Add --group command line option.
        * Scripts/webkitpy/tool/steps/runtests.py:
        (RunTests.options): Add group.
        (RunTests.run): Check for JSC.
        (RunTests._run_webkit_tests): Check for JSC.
        (RunTests): Add _group attribute.
        (RunTests._run_javascriptcore_tests): New.
        * Scripts/webkitpy/tool/steps/steps_unittest.py: Unit tests.

2016-09-25  Gyuyoung Kim  <gyuyoung.kim@webkit.org>

        [EFL] Adjust EFL indentation to EFL MiniBrowser



# Copyright (C) 2011 Google Inc. All rights reserved.
# Copyright (C) 2017 Apple Inc. All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are

        self.added_paths = set()
        self._filesystem = filesystem or MockFileSystem()
        self._executive = executive or MockExecutive()
        self._mockChangedFiles = ["MockFile1"]

    def add(self, destination_path):
        self.add_list([destination_path])

        return self._filesystem.join(self.checkout_root, *comps)

    def changed_files(self, git_commit=None):
        return self._mockChangedFiles

    def changed_files_for_revision(self, revision):
        return ["MockFile1"]

Lines 50-54 a/Tools/Scripts/webkitpy/common/config/ews.json_sec1
- a/Tools/Scripts/webkitpy/common/config/ews.json +6 lines
50	"port": "mac",	50	"port": "mac",
51	"name": "mac-32bit-ews",	51	"name": "mac-32bit-ews",
52	"architecture": "i386"	52	"architecture": "i386"
		53	},
		54	"JSC EWS": {
		55	"port": "mac",
		56	"name": "jsc-ews",
		57	"group": "jsc",
		58	"runTests": true
53	}	59	}
54	}	60	}



# Copyright (C) 2009, Google Inc. All rights reserved.
# Copyright (C) 2013 Nokia Corporation and/or its subsidiary(-ies).
# Copyright (C) 2015, 2017 Apple Inc. All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are

            command.append("--release")
        return command

    def build_jsc_command(self, build_style=None):
        command = self.script_shell_command("build-jsc")
        if build_style == "debug":
            command.append("--debug")
        if build_style == "release":
            command.append("--release")
        return command

    def run_javascriptcore_tests_command(self, build_style=None):
        command = self.script_shell_command("run-javascriptcore-tests")
        command.append("--no-fail-fast")
        command.append("--json-output=jsc_test_results.json")
        if (build_style == "debug"):
            command.append("--debug")
        return command

    def run_webkit_tests_command(self, build_style=None):
        command = self.script_shell_command("run-webkit-tests")

class MacPort(DeprecatedPort):
    port_flag_name = "mac"

    def run_webkit_tests_command(self, build_style=None, test_suite=None):
        if (test_suite == "jsc"):
            return super(MacPort, self).run_javascriptcore_tests_command(build_style)

        command = super(MacPort, self).run_webkit_tests_command(build_style)
        command.append("--dump-render-tree")
        return command



# Copyright (C) 2011 Google Inc. All rights reserved.
# Copyright (C) 2017 Apple Inc. All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are

    def run_perl_unittests_command(self):
        return ['mock-test-webkitperl']

    def run_javascriptcore_tests_command(self, build_style=None):
        return ['mock-run-javacriptcore-tests']

    def run_webkit_tests_command(self, build_style=None):

Lines 1-4 a/Tools/Scripts/webkitpy/common/config/ports_unittest.py_sec1
- a/Tools/Scripts/webkitpy/common/config/ports_unittest.py +5 lines
1	# Copyright (c) 2009, Google Inc. All rights reserved.	1	# Copyright (c) 2009, Google Inc. All rights reserved.
		2	# Copyright (C) 2017 Apple Inc. All rights reserved.
2	#	3	#
3	# Redistribution and use in source and binary forms, with or without	4	# Redistribution and use in source and binary forms, with or without
4	# modification, are permitted provided that the following conditions are	5	# modification, are permitted provided that the following conditions are
Lines 39-44 class DeprecatedPortTest(unittest.TestCase): a/Tools/Scripts/webkitpy/common/config/ports_unittest.py_sec2
39	self.assertEqual(MacPort().build_webkit_command(build_style="debug"), DeprecatedPort().script_shell_command("build-webkit") + ["--debug"])	40	self.assertEqual(MacPort().build_webkit_command(build_style="debug"), DeprecatedPort().script_shell_command("build-webkit") + ["--debug"])
40	self.assertEqual(MacPort().build_webkit_command(build_style="release"), DeprecatedPort().script_shell_command("build-webkit") + ["--release"])	41	self.assertEqual(MacPort().build_webkit_command(build_style="release"), DeprecatedPort().script_shell_command("build-webkit") + ["--release"])
41		42
		43	self.assertEqual(MacPort().build_jsc_command(), DeprecatedPort().script_shell_command("build-jsc"))
		44	self.assertEqual(MacPort().build_jsc_command(build_style="release"), DeprecatedPort().script_shell_command("build-jsc") + ["--release"])
		45	self.assertEqual(MacPort().build_jsc_command(build_style="debug"), DeprecatedPort().script_shell_command("build-jsc") + ["--debug"])
		46
42	def test_gtk_wk2_port(self):	47	def test_gtk_wk2_port(self):
43	self.assertEqual(GtkWK2Port().flag(), "--port=gtk-wk2")	48	self.assertEqual(GtkWK2Port().flag(), "--port=gtk-wk2")
44	self.assertEqual(GtkWK2Port().run_webkit_tests_command(), DeprecatedPort().script_shell_command("run-webkit-tests") + ["--gtk"])	49	self.assertEqual(GtkWK2Port().run_webkit_tests_command(), DeprecatedPort().script_shell_command("run-webkit-tests") + ["--gtk"])



# Copyright (C) 2017 Apple Inc. All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions
# are met:
# 1.  Redistributions of source code must retain the above copyright
#     notice, this list of conditions and the following disclaimer.
# 2.  Redistributions in binary form must reproduce the above copyright
#     notice, this list of conditions and the following disclaimer in the
#     documentation and/or other materials provided with the distribution.
#
# THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS'' AND
# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
# DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS BE LIABLE FOR
# ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
# SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

import json
import logging

_log = logging.getLogger(__name__)


class JSCTestResults(object):
    @classmethod
    def intersection(cls, first, second):
        intersection_api_tests_passed = first._all_api_tests_passed or second._all_api_tests_passed
        intersection_stress_test_failures = list(set(first._stress_test_failures) & set(second._stress_test_failures))
        return cls(intersection_api_tests_passed, intersection_stress_test_failures)

    @classmethod
    def results_from_string(cls, string):
        if not string:
            return None

        try:
            parsed_results = json.loads(string)
        except ValueError:
            return None

        if 'allApiTestsPassed' not in parsed_results or 'stressTestFailures' not in parsed_results:
            return None

        return cls(parsed_results['allApiTestsPassed'], parsed_results['stressTestFailures'])

    def __init__(self, all_api_tests_passed, stress_test_failures):
        self._all_api_tests_passed = all_api_tests_passed
        self._stress_test_failures = stress_test_failures

        self._test_results = stress_test_failures[:]
        if not self._all_api_tests_passed:
            self._test_results.append("apiTests")

    def equals(self, other):
        return self._all_api_tests_passed == other._all_api_tests_passed and \
               set(self._stress_test_failures) == set(other._stress_test_failures)

    def is_subset(self, other):
        return set(self.test_results()) <= set(other.test_results())

    def stress_test_failures(self):
        return self._stress_test_failures

    def all_api_tests_passed(self):
        return self._all_api_tests_passed

    def all_passed(self):
        return self._all_api_tests_passed and not self._stress_test_failures

    def failing_test_results(self):
        return self.stress_test_failures()

    def test_results(self):
        return self._test_results

    def failing_tests(self):
        return self._test_results

    # No defined failure limit for JSC.
    def did_exceed_test_failure_limit(self):
        return False



# Copyright (C) 2017 Apple Inc. All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions
# are met:
# 1.  Redistributions of source code must retain the above copyright
#     notice, this list of conditions and the following disclaimer.
# 2.  Redistributions in binary form must reproduce the above copyright
#     notice, this list of conditions and the following disclaimer in the
#     documentation and/or other materials provided with the distribution.
#
# THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS'' AND
# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
# DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS BE LIABLE FOR
# ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
# SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

import unittest

from webkitpy.common.net.jsctestresults import JSCTestResults


class JSCTestResultsTest(unittest.TestCase):
    def test_results_from_string(self):
        none_item = None
        empty_string = ""
        invalid_json = '{"allApiTestsPassed":'
        incomplete_json_v1 = '{"allApiTestsPassed": true}'
        incomplete_json_v2 = '{"stressTestFailures":[]}'
        self.assertEqual(None, JSCTestResults.results_from_string(none_item))
        self.assertEqual(None, JSCTestResults.results_from_string(empty_string))
        self.assertEqual(None, JSCTestResults.results_from_string(invalid_json))
        self.assertEqual(None, JSCTestResults.results_from_string(incomplete_json_v1))
        self.assertEqual(None, JSCTestResults.results_from_string(incomplete_json_v2))

        no_failures_string = '{"allApiTestsPassed": true, "stressTestFailures":[]}'
        no_failures_results = JSCTestResults(True, [])
        self.assertTrue(no_failures_results.equals(JSCTestResults.results_from_string(no_failures_string)))

        api_test_failures_string = '{"allApiTestsPassed": false, "stressTestFailures":[]}'
        api_test_failures_results = JSCTestResults(False, [])
        self.assertTrue(api_test_failures_results.equals(JSCTestResults.results_from_string(api_test_failures_string)))

        many_failures_string = '{"allApiTestsPassed": false, "stressTestFailures":["es6.yaml/es6/typed_arrays_Int16Array.js.default", "es6.yaml/es6/typed_arrays_Int8Array.js.default"]}'
        many_failures_results = JSCTestResults(False, ["es6.yaml/es6/typed_arrays_Int16Array.js.default", "es6.yaml/es6/typed_arrays_Int8Array.js.default"])
        self.assertTrue(many_failures_results.equals(JSCTestResults.results_from_string(many_failures_string)))

        self.assertFalse(no_failures_results == api_test_failures_results)
        self.assertFalse(api_test_failures_results == many_failures_results)

    def test_intersection_api_tests(self):
        results1 = JSCTestResults(False, [])
        results2 = JSCTestResults(True, [])

        expected_intersection = JSCTestResults(True, [])
        self.assertTrue(expected_intersection.equals(JSCTestResults.intersection(results1, results2)))

    def test_intersection_stress_tests(self):
        results1 = JSCTestResults(True, ['failure1', 'failure2'])
        results2 = JSCTestResults(True, ['failure1', 'failure3'])

        expected_intersection = JSCTestResults(True, ['failure1'])
        self.assertTrue(expected_intersection.equals(JSCTestResults.intersection(results1, results2)))

    def test_intersection_general_case(self):
        results1 = JSCTestResults(True, ['failure1', 'failure2'])
        results2 = JSCTestResults(False, ['failure1'])

        expected_intersection = JSCTestResults(True, ['failure1'])
        self.assertTrue(expected_intersection.equals(JSCTestResults.intersection(results1, results2)))

Lines 1-4 a/Tools/Scripts/webkitpy/tool/bot/earlywarningsystemtask.py_sec1
- a/Tools/Scripts/webkitpy/tool/bot/earlywarningsystemtask.py -1 / +4 lines
1	# Copyright (c) 2011 Google Inc. All rights reserved.	1	# Copyright (c) 2011 Google Inc. All rights reserved.
		2	# Copyright (C) 2017 Apple Inc. All rights reserved.
2	#	3	#
3	# Redistribution and use in source and binary forms, with or without	4	# Redistribution and use in source and binary forms, with or without
4	# modification, are permitted provided that the following conditions are	5	# modification, are permitted provided that the following conditions are
Lines 26-32 a/Tools/Scripts/webkitpy/tool/bot/earlywarningsystemtask.py_sec2
26	# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE	27	# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
27	# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.	28	# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
28		29
29	from webkitpy.tool.bot.patchanalysistask import PatchAnalysisTask, PatchAnalysisTaskDelegate, UnableToApplyPatch, PatchIsNotValid	30	from webkitpy.tool.bot.patchanalysistask import PatchAnalysisTask, PatchAnalysisTaskDelegate, UnableToApplyPatch, PatchIsNotValid, PatchIsNotApplicable
30		31
31		32
32	class EarlyWarningSystemTaskDelegate(PatchAnalysisTaskDelegate):	33	class EarlyWarningSystemTaskDelegate(PatchAnalysisTaskDelegate):
Lines 58-63 class EarlyWarningSystemTask(PatchAnalysisTask): a/Tools/Scripts/webkitpy/tool/bot/earlywarningsystemtask.py_sec3
58	return False	59	return False
59	if not self._apply():	60	if not self._apply():
60	raise UnableToApplyPatch(self._patch)	61	raise UnableToApplyPatch(self._patch)
		62	if not self._check_patch_relevance():
		63	raise PatchIsNotApplicable(self._patch)
61	if not self._build():	64	if not self._build():
62	if not self._build_without_patch():	65	if not self._build_without_patch():
63	return False	66	return False



# Copyright (C) 2017 Apple Inc. All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions
# are met:
# 1.  Redistributions of source code must retain the above copyright
#     notice, this list of conditions and the following disclaimer.
# 2.  Redistributions in binary form must reproduce the above copyright
#     notice, this list of conditions and the following disclaimer in the
#     documentation and/or other materials provided with the distribution.
#
# THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS'' AND
# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
# DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS BE LIABLE FOR
# ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
# SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

import logging
import unittest

from webkitpy.common.net.jsctestresults import JSCTestResults
from webkitpy.common.system.executive import ScriptError
from webkitpy.tool.bot.patchanalysistask import *
from webkitpy.tool.commands.earlywarningsystem import AbstractEarlyWarningSystem
from webkitpy.tool.mocktool import MockTool

_log = logging.getLogger(__name__)


class MockPatchAnalysisTask(PatchAnalysisTask):
    def __init__(self, delegate, patch, patches_passed_all_tests):
        self._delegate = delegate
        self._patch = patch
        self._script_error = None
        self._results_archive_from_patch_test_run = None
        self._results_from_patch_test_run = None
        self.error = None
        self._patches_passed_all_tests = patches_passed_all_tests
        self._test_run_count = 0
        self.failure_status_id = 42

    def _test(self):
        self._test_run_count += 1

        if self._patches_passed_all_tests.pop() == True:
            return True
        else:
            self._script_error = ScriptError("Regression test")
            return False

    def _build_and_test_without_patch(self):
        self._test_run_count += 1
        return True

    def validate(self):
        return True

    def test_run_count(self):
        return self._test_run_count


# This is the delegate to be used with MockPatchAnalysisTask, above.
class MockJSCEarlyWarningSystem(AbstractEarlyWarningSystem):
    def __init__(self, first_test_results, second_test_results, clean_test_results):
        AbstractEarlyWarningSystem.__init__(self)
        self._group = "jsc"
        self._results_in_order = [clean_test_results, second_test_results, first_test_results]

    def test_results(self):
        return self._results_in_order.pop()


class JSCEarlyWarningSystemTest(unittest.TestCase):
    def _results_indicate_all_passed(self, results):
        if results == None:
            return False
        return results.all_passed()

    def _create_task(self, first_test_results, second_test_results, clean_test_results):
        queue = MockJSCEarlyWarningSystem(first_test_results, second_test_results, clean_test_results)
        tool = MockTool(log_executive=True)
        patch = tool.bugs.fetch_attachment(10000)
        patches_passed_all_tests = map(self._results_indicate_all_passed, [second_test_results, first_test_results])
        return MockPatchAnalysisTask(queue, patch, patches_passed_all_tests)

    def test_success_case(self):
        first_test_results = JSCTestResults(True, [])
        second_test_results = JSCTestResults(True, [])
        clean_test_results = JSCTestResults(True, [])
        task = self._create_task(first_test_results, second_test_results, clean_test_results)

        return_value = task._test_patch()
        self.assertEqual(task.test_run_count(), 1)
        self.assertEqual(return_value, True)

    def test_test_failure(self):
        first_test_results = JSCTestResults(True, ['Fail.js'])
        second_test_results = JSCTestResults(True, ['Fail.js'])
        clean_test_results = JSCTestResults(True, [])
        task = self._create_task(first_test_results, second_test_results, clean_test_results)

        with self.assertRaises(ScriptError):
            return_value = task._test_patch()
        self.assertEqual(task.test_run_count(), 3)

    def test_fix(self):
        first_test_results = JSCTestResults(True, [])
        second_test_results = JSCTestResults(True, [])
        clean_test_results = JSCTestResults(True, ['Fail.js'])
        task = self._create_task(first_test_results, second_test_results, clean_test_results)

        return_value = task._test_patch()
        self.assertEqual(task.test_run_count(), 1)
        self.assertEqual(return_value, True)

    def test_ineffective_patch(self):
        first_test_results = JSCTestResults(False, ['failure1.js', 'failure2.js'])
        second_test_results = JSCTestResults(False, ['failure1.js', 'failure2.js'])
        clean_test_results = JSCTestResults(False, ['failure1.js', 'failure2.js'])
        task = self._create_task(first_test_results, second_test_results, clean_test_results)

        return_value = task._test_patch()
        self.assertEqual(task.test_run_count(), 3)
        self.assertEqual(return_value, True)

    def test_partially_effective_patch(self):
        first_test_results = JSCTestResults(True, ['failure2.js'])
        second_test_results = JSCTestResults(True, ['failure2.js'])
        clean_test_results = JSCTestResults(False, ['failure1.js', 'failure2.js'])
        task = self._create_task(first_test_results, second_test_results, clean_test_results)

        return_value = task._test_patch()
        self.assertEqual(task.test_run_count(), 3)
        self.assertEqual(return_value, True)

    def test_different_test_failures_in_patch_and_tree(self):
        first_test_results = JSCTestResults(False, [])
        second_test_results = JSCTestResults(False, [])
        clean_test_results = JSCTestResults(True, ['failure1.js', 'failure2.js'])
        task = self._create_task(first_test_results, second_test_results, clean_test_results)

        with self.assertRaises(ScriptError):
            return_value = task._test_patch()
        self.assertEqual(task.test_run_count(), 3)

    def test_first_results_could_not_be_read(self):
        first_test_results = None
        second_test_results = JSCTestResults(True, [])
        clean_test_results = JSCTestResults(True, [])
        task = self._create_task(first_test_results, second_test_results, clean_test_results)

        with self.assertRaises(ScriptError):
            return_value = task._test_patch()
        self.assertEqual(task.test_run_count(), 1)

    def test_second_results_could_not_be_read(self):
        first_test_results = JSCTestResults(False, ['failure1.js', 'failure2.js'])
        second_test_results = None
        clean_test_results = JSCTestResults(True, [])
        task = self._create_task(first_test_results, second_test_results, clean_test_results)

        with self.assertRaises(ScriptError):
            return_value = task._test_patch()
        self.assertEqual(task.test_run_count(), 2)

    def test_clean_results_could_not_be_read(self):
        first_test_results = JSCTestResults(True, ['failure2.js'])
        second_test_results = JSCTestResults(True, ['failure2.js'])
        clean_test_results = None
        task = self._create_task(first_test_results, second_test_results, clean_test_results)

        with self.assertRaises(ScriptError):
            return_value = task._test_patch()
        self.assertEqual(task.test_run_count(), 3)

    def test_flaky_results_on_clean_tree_pass(self):
        first_test_results = JSCTestResults(True, ['failure2.js'])
        second_test_results = JSCTestResults(True, [])
        clean_test_results = JSCTestResults(True, [])
        task = self._create_task(first_test_results, second_test_results, clean_test_results)

        return_value = task._test_patch()
        self.assertEqual(return_value, True)
        self.assertEqual(task.test_run_count(), 2)

    def test_flaky_results_on_clean_tree_pass_v2(self):
        first_test_results = JSCTestResults(True, [])
        second_test_results = JSCTestResults(True, ['failure2.js'])
        clean_test_results = JSCTestResults(True, [])
        task = self._create_task(first_test_results, second_test_results, clean_test_results)

        return_value = task._test_patch()
        self.assertEqual(return_value, True)
        self.assertEqual(task.test_run_count(), 1)

    def test_flaky_results_on_clean_tree_failure(self):
        first_test_results = JSCTestResults(False, ['failure1.js', 'failure2.js'])
        second_test_results = JSCTestResults(True, ['failure2.js'])
        clean_test_results = JSCTestResults(True, [])
        task = self._create_task(first_test_results, second_test_results, clean_test_results)

        with self.assertRaises(ScriptError):
            return_value = task._test_patch()
        self.assertEqual(task.test_run_count(), 3)

    def test_flaky_results_on_red_tree_pass(self):
        first_test_results = JSCTestResults(True, ['failure1.js'])
        second_test_results = JSCTestResults(True, ['failure1.js', 'failure2.js'])
        clean_test_results = JSCTestResults(True, ['failure1.js'])
        task = self._create_task(first_test_results, second_test_results, clean_test_results)

        return_value = task._test_patch()
        self.assertEqual(return_value, True)
        self.assertEqual(task.test_run_count(), 3)



# Copyright (C) 2017 Apple Inc. All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions
# are met:
# 1.  Redistributions of source code must retain the above copyright
#     notice, this list of conditions and the following disclaimer.
# 2.  Redistributions in binary form must reproduce the above copyright
#     notice, this list of conditions and the following disclaimer in the
#     documentation and/or other materials provided with the distribution.
#
# THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS'' AND
# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
# DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS BE LIABLE FOR
# ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
# SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

import logging

from webkitpy.common.net.jsctestresults import JSCTestResults

_log = logging.getLogger(__name__)


class JSCTestResultsReader(object):
    def __init__(self, host, results_directory):
        self._host = host
        self._results_directory = results_directory

    def _read_file_contents(self, path):
        try:
            return self._host.filesystem.read_text_file(path)
        except IOError, e:  # File does not exist or can't be read.
            return None

    def _create_jsc_test_results(self):
        results_path = self._host.filesystem.join(self._results_directory, "jsc_test_results.json")
        contents = self._read_file_contents(results_path)
        return JSCTestResults.results_from_string(contents)

    def results(self):
        return self._create_jsc_test_results()

Lines 1-4 a/Tools/Scripts/webkitpy/tool/bot/patchanalysistask.py_sec1
- a/Tools/Scripts/webkitpy/tool/bot/patchanalysistask.py -62 / +135 lines
1	# Copyright (c) 2010 Google Inc. All rights reserved.	1	# Copyright (c) 2010 Google Inc. All rights reserved.
		2	# Copyright (C) 2017 Apple Inc. All rights reserved.
2	#	3	#
3	# Redistribution and use in source and binary forms, with or without	4	# Redistribution and use in source and binary forms, with or without
4	# modification, are permitted provided that the following conditions are	5	# modification, are permitted provided that the following conditions are
Lines 28-33 a/Tools/Scripts/webkitpy/tool/bot/patchanalysistask.py_sec2
28		29
29	from webkitpy.common.system.executive import ScriptError	30	from webkitpy.common.system.executive import ScriptError
30	from webkitpy.common.net.layouttestresults import LayoutTestResults	31	from webkitpy.common.net.layouttestresults import LayoutTestResults
		32	from webkitpy.common.net.jsctestresults import JSCTestResults
31		33
32		34
33	class UnableToApplyPatch(Exception):	35	class UnableToApplyPatch(Exception):
Lines 43-48 class PatchIsNotValid(Exception): a/Tools/Scripts/webkitpy/tool/bot/patchanalysistask.py_sec3
43	self.failure_message = failure_message	45	self.failure_message = failure_message
44		46
45		47
		48	class PatchIsNotApplicable(Exception):
		49	def __init__(self, patch):
		50	Exception.__init__(self)
		51	self.patch = patch
		52
46	class PatchAnalysisTaskDelegate(object):	53	class PatchAnalysisTaskDelegate(object):
47	def parent_command(self):	54	def parent_command(self):
48	raise NotImplementedError("subclasses must implement")	55	raise NotImplementedError("subclasses must implement")
Lines 122-149 class PatchAnalysisTask(object): a/Tools/Scripts/webkitpy/tool/bot/patchanalysistask.py_sec4
122	"Applied patch",	129	"Applied patch",
123	"Patch does not apply")	130	"Patch does not apply")
124		131
		132	def _check_patch_relevance(self):
		133	args = [
		134	"check-patch-relevance",
		135	]
		136
		137	if hasattr(self._delegate, 'group'):
		138	args.append("--group=%s" % self._delegate.group())
		139
		140	return self._run_command(
		141	args,
		142	"Checked relevance of patch",
		143	"Patch was not relevant")
		144
125	def _build(self):	145	def _build(self):
126	return self._run_command([	146	args = [
127	"build",	147	"build",
128	"--no-clean",	148	"--no-clean",
129	"--no-update",	149	"--no-update",
130	"--build-style=%s" % self._delegate.build_style(),	150	"--build-style=%s" % self._delegate.build_style(),
131	],	151	]
132	"Built patch",	152
133	"Patch does not build")	153	if hasattr(self._delegate, 'group'):
		154	args.append("--group=%s" % self._delegate.group())
		155
		156	return self._run_command(
		157	args,
		158	"Built patch",
		159	"Patch does not build")
134		160
135	def _build_without_patch(self):	161	def _build_without_patch(self):
136	return self._run_command([	162	args = [
137	"build",	163	"build",
138	"--force-clean",	164	"--force-clean",
139	"--no-update",	165	"--no-update",
140	"--build-style=%s" % self._delegate.build_style(),	166	"--build-style=%s" % self._delegate.build_style(),
141	],	167	]
142	"Able to build without patch",	168
143	"Unable to build without patch")	169	if hasattr(self._delegate, 'group'):
		170	args.append("--group=%s" % self._delegate.group())
		171
		172	return self._run_command(
		173	args,
		174	"Able to build without patch",
		175	"Unable to build without patch")
144		176
145	def _test(self):	177	def _test(self):
146	return self._run_command([	178	args = [
147	"build-and-test",	179	"build-and-test",
148	"--no-clean",	180	"--no-clean",
149	"--no-update",	181	"--no-update",
Lines 151-162 class PatchAnalysisTask(object): a/Tools/Scripts/webkitpy/tool/bot/patchanalysistask.py_sec5
151	"--test",	183	"--test",
152	"--non-interactive",	184	"--non-interactive",
153	"--build-style=%s" % self._delegate.build_style(),	185	"--build-style=%s" % self._delegate.build_style(),
154	],	186	]
155	"Passed tests",	187
156	"Patch does not pass tests")	188	if hasattr(self._delegate, 'group'):
		189	args.append("--group=%s" % self._delegate.group())
		190
		191	return self._run_command(
		192	args,
		193	"Passed tests",
		194	"Patch does not pass tests")
157		195
158	def _build_and_test_without_patch(self):	196	def _build_and_test_without_patch(self):
159	return self._run_command([	197	args = [
160	"build-and-test",	198	"build-and-test",
161	"--force-clean",	199	"--force-clean",
162	"--no-update",	200	"--no-update",
Lines 164-172 class PatchAnalysisTask(object): a/Tools/Scripts/webkitpy/tool/bot/patchanalysistask.py_sec6
164	"--test",	202	"--test",
165	"--non-interactive",	203	"--non-interactive",
166	"--build-style=%s" % self._delegate.build_style(),	204	"--build-style=%s" % self._delegate.build_style(),
167	],	205	]
168	"Able to pass tests without patch",	206
169	"Unable to pass tests without patch (tree is red?)")	207	if hasattr(self._delegate, 'group'):
		208	args.append("--group=%s" % self._delegate.group())
		209
		210	return self._run_command(
		211	args,
		212	"Able to pass tests without patch",
		213	"Unable to pass tests without patch (tree is red?)")
170		214
171	def _land(self):	215	def _land(self):
172	# Unclear if this should pass --quiet or not. If --parent-command always does the reporting, then it should.	216	# Unclear if this should pass --quiet or not. If --parent-command always does the reporting, then it should.
Lines 210-278 class PatchAnalysisTask(object): a/Tools/Scripts/webkitpy/tool/bot/patchanalysistask.py_sec7
210	if self._test():	254	if self._test():
211	return True	255	return True
212		256
213	# Note: archive_last_test_results deletes the results directory, making these calls order-sensitve.	257	# FIXME: Abstract out the retry logic instead of casing on the test suite here.
214	# We could remove this dependency by building the test_results from the archive.	258	if hasattr(self._delegate, 'group') and self._delegate.group() == "jsc":
215	first_results = self._delegate.test_results()	259	first_results = self._delegate.test_results()
216	first_results_archive = self._delegate.archive_last_test_results(self._patch)	260	first_script_error = self._script_error
217	first_script_error = self._script_error	261	first_failure_status_id = self.failure_status_id
218	first_failure_status_id = self.failure_status_id	262	if first_results is None:
219		263	return self.report_failure(None, None, ScriptError("Could not read JSC test results for patch."))
220	if self._test() and not first_results.did_exceed_test_failure_limit():
221	# Only report flaky tests if we were successful at parsing results.json and archiving results.
222	if first_results and first_results_archive:
223	self._report_flaky_tests(first_results.failing_test_results(), first_results_archive)
224	return True
225		264
226	second_results = self._delegate.test_results()	265	if self._test():
227	second_results_archive = self._delegate.archive_last_test_results(self._patch)	266	return True
228	second_script_error = self._script_error	267	second_results = self._delegate.test_results()
229	second_failure_status_id = self.failure_status_id	268	second_script_error = self._script_error
		269	if second_results is None:
		270	return self.report_failure(None, None, ScriptError("Could not read JSC test results on second run."))
		271
		272	consistent_results = JSCTestResults.intersection(first_results, second_results)
230		273
231	if second_results.did_exceed_test_failure_limit() and first_results.did_exceed_test_failure_limit():
232	self._build_and_test_without_patch()	274	self._build_and_test_without_patch()
233	clean_tree_results = self._delegate.test_results()	275	clean_tree_results = self._delegate.test_results()
		276	if clean_tree_results is None:
		277	return self.report_failure(None, None, ScriptError("Could not read JSC test results for clean tree."))
234		278
235	if (len(first_results.failing_tests()) - len(clean_tree_results.failing_tests())) <= 5:	279	if consistent_results.is_subset(clean_tree_results):
236	return False	280	return True
237		281
238	self.failure_status_id = first_failure_status_id	282	self.failure_status_id = first_failure_status_id
		283	return self.report_failure(None, consistent_results, first_script_error)
239		284
240	return self.report_failure(first_results_archive, first_results, first_script_error)	285	else:
		286	# Note: archive_last_test_results deletes the results directory, making these calls order-sensitve.
		287	# We could remove this dependency by building the test_results from the archive.
		288	first_results = self._delegate.test_results()
		289	first_results_archive = self._delegate.archive_last_test_results(self._patch)
		290	first_script_error = self._script_error
		291	first_failure_status_id = self.failure_status_id
241		292
242	if second_results.did_exceed_test_failure_limit():	293	if self._test() and not first_results.did_exceed_test_failure_limit():
243	self._should_defer_patch_or_throw(first_results.failing_test_results(), first_results_archive, first_script_error, first_failure_status_id)	294	# Only report flaky tests if we were successful at parsing results.json and archiving results.
244	return False	295	if first_results and first_results_archive:
		296	self._report_flaky_tests(first_results.failing_test_results(), first_results_archive)
		297	return True
245		298
246	if first_results.did_exceed_test_failure_limit():	299	second_results = self._delegate.test_results()
247	self._should_defer_patch_or_throw(second_results.failing_test_results(), second_results_archive, second_script_error, second_failure_status_id)	300	second_results_archive = self._delegate.archive_last_test_results(self._patch)
248	return False	301	second_script_error = self._script_error
		302	second_failure_status_id = self.failure_status_id
249		303
250	if self._results_failed_different_tests(first_results, second_results):	304	if second_results.did_exceed_test_failure_limit() and first_results.did_exceed_test_failure_limit():
251	first_failing_results_set = frozenset(first_results.failing_test_results())	305	self._build_and_test_without_patch()
252	second_failing_results_set = frozenset(second_results.failing_test_results())	306	clean_tree_results = self._delegate.test_results()
253		307
254	tests_that_only_failed_first = first_failing_results_set.difference(second_failing_results_set)	308	if (len(first_results.failing_tests()) - len(clean_tree_results.failing_tests())) <= 5:
255	self._report_flaky_tests(tests_that_only_failed_first, first_results_archive)	309	return False
256		310
257	tests_that_only_failed_second = second_failing_results_set.difference(first_failing_results_set)	311	self.failure_status_id = first_failure_status_id
258	self._report_flaky_tests(tests_that_only_failed_second, second_results_archive)
259		312
260	tests_that_consistently_failed = first_failing_results_set.intersection(second_failing_results_set)	313	return self.report_failure(first_results_archive, first_results, first_script_error)
261	if tests_that_consistently_failed:
262	if self._should_defer_patch_or_throw(tests_that_consistently_failed, first_results_archive, first_script_error, first_failure_status_id):
263	return False # Defer patch
264		314
265	# At this point we know that at least one test flaked, but no consistent failures	315	if second_results.did_exceed_test_failure_limit():
266	# were introduced. This is a bit of a grey-zone.	316	self._should_defer_patch_or_throw(first_results.failing_test_results(), first_results_archive, first_script_error, first_failure_status_id)
267	return False # Defer patch	317	return False
		318
		319	if first_results.did_exceed_test_failure_limit():
		320	self._should_defer_patch_or_throw(second_results.failing_test_results(), second_results_archive, second_script_error, second_failure_status_id)
		321	return False
		322
		323	if self._results_failed_different_tests(first_results, second_results):
		324	first_failing_results_set = frozenset(first_results.failing_test_results())
		325	second_failing_results_set = frozenset(second_results.failing_test_results())
		326
		327	tests_that_only_failed_first = first_failing_results_set.difference(second_failing_results_set)
		328	self._report_flaky_tests(tests_that_only_failed_first, first_results_archive)
268		329
269	if self._should_defer_patch_or_throw(first_results.failing_test_results(), first_results_archive, first_script_error, first_failure_status_id):	330	tests_that_only_failed_second = second_failing_results_set.difference(first_failing_results_set)
270	return False # Defer patch	331	self._report_flaky_tests(tests_that_only_failed_second, second_results_archive)
271		332
272	# At this point, we know that the first and second runs had the exact same failures,	333	tests_that_consistently_failed = first_failing_results_set.intersection(second_failing_results_set)
273	# and that those failures are all present on the clean tree, so we can say with certainty	334	if tests_that_consistently_failed:
274	# that the patch is good.	335	if self._should_defer_patch_or_throw(tests_that_consistently_failed, first_results_archive, first_script_error, first_failure_status_id):
275	return True	336	return False # Defer patch
		337
		338	# At this point we know that at least one test flaked, but no consistent failures
		339	# were introduced. This is a bit of a grey-zone.
		340	return False # Defer patch
		341
		342	if self._should_defer_patch_or_throw(first_results.failing_test_results(), first_results_archive, first_script_error, first_failure_status_id):
		343	return False # Defer patch
		344
		345	# At this point, we know that the first and second runs had the exact same failures,
		346	# and that those failures are all present on the clean tree, so we can say with certainty
		347	# that the patch is good.
		348	return True
276		349
277	def results_archive_from_patch_test_run(self, patch):	350	def results_archive_from_patch_test_run(self, patch):
278	assert(self._patch.id() == patch.id()) # PatchAnalysisTask is not currently re-useable.	351	assert(self._patch.id() == patch.id()) # PatchAnalysisTask is not currently re-useable.

Lines 1-5 a/Tools/Scripts/webkitpy/tool/commands/download.py_sec1
- a/Tools/Scripts/webkitpy/tool/commands/download.py -1 / +9 lines
1	# Copyright (c) 2009, 2011 Google Inc. All rights reserved.	1	# Copyright (c) 2009, 2011 Google Inc. All rights reserved.
2	# Copyright (c) 2009 Apple Inc. All rights reserved.	2	# Copyright (c) 2009, 2017 Apple Inc. All rights reserved.
3	#	3	#
4	# Redistribution and use in source and binary forms, with or without	4	# Redistribution and use in source and binary forms, with or without
5	# modification, are permitted provided that the following conditions are	5	# modification, are permitted provided that the following conditions are
Lines 87-92 class BuildAndTest(AbstractSequencedCommand): a/Tools/Scripts/webkitpy/tool/commands/download.py_sec2
87	]	87	]
88		88
89		89
		90	class CheckPatchRelevance(AbstractSequencedCommand):
		91	name = "check-patch-relevance"
		92	help_text = "Check if this patch needs to be tested"
		93	steps = [
		94	steps.CheckPatchRelevance,
		95	]
		96
		97
90	class Land(AbstractSequencedCommand):	98	class Land(AbstractSequencedCommand):
91	name = "land"	99	name = "land"
92	help_text = "Land the current working directory diff and updates the associated bug if any"	100	help_text = "Land the current working directory diff and updates the associated bug if any"



# Copyright (c) 2009 Google Inc. All rights reserved.
# Copyright (c) 2017 Apple Inc. All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are


import json
import logging
import os
from optparse import make_option

from webkitpy.common.config.committers import CommitterList

from webkitpy.common.system.executive import ScriptError
from webkitpy.tool.bot.earlywarningsystemtask import EarlyWarningSystemTask, EarlyWarningSystemTaskDelegate
from webkitpy.tool.bot.layouttestresultsreader import LayoutTestResultsReader
from webkitpy.tool.bot.jsctestresultsreader import JSCTestResultsReader
from webkitpy.tool.bot.patchanalysistask import UnableToApplyPatch, PatchIsNotValid, PatchIsNotApplicable
from webkitpy.tool.bot.queueengine import QueueEngine
from webkitpy.tool.commands.queues import AbstractReviewQueue



    def begin_work_queue(self):
        AbstractReviewQueue.begin_work_queue(self)

        if self.group() == "jsc":
            dir = os.path.dirname(os.path.dirname(os.path.dirname(self._port.results_directory())))
            self._test_results_reader = JSCTestResultsReader(self._tool, dir)
        else:
            self._test_results_reader = LayoutTestResultsReader(self._tool, self._port.results_directory(), self._log_directory())

    def _failing_tests_message(self, task, patch):
        results = task.results_from_patch_test_run(patch)

        except UnableToApplyPatch, e:
            self._did_error(patch, "%s unable to apply patch." % self.name)
            return False
        except PatchIsNotApplicable, e:
            self._did_error(patch, "%s patch not applicable." % self.name)
            return False
        except ScriptError, e:
            self._post_reject_message_on_bug(self._tool, patch, task.failure_status_id, self._failing_tests_message(task, patch))
            results_archive = task.results_archive_from_patch_test_run(patch)

        return self._update_status(message, patch=patch, results_file=failure_log)

    def test_results(self):
        return self._test_results_reader.results()

    def archive_last_test_results(self, patch):
        return self._test_results_reader.archive(patch)

    def build_style(self):
        return self._build_style

    def group(self):
        return self._group

    def refetch_patch(self, patch):
        return self._tool.bugs.fetch_attachment(patch.id())


                '_build_style': config.get('style', "release"),
                'watchers': config.get('watchers', []),
                'run_tests': config.get('runTests', cls.run_tests),
                '_group': config.get('group', None),
            }))
        return classes

Lines 1-4 a/Tools/Scripts/webkitpy/tool/commands/earlywarningsystem_unittest.py_sec1
- a/Tools/Scripts/webkitpy/tool/commands/earlywarningsystem_unittest.py -2 / +24 lines
1	# Copyright (C) 2009 Google Inc. All rights reserved.	1	# Copyright (C) 2009 Google Inc. All rights reserved.
		2	# Copyright (C) 2017 Apple Inc. All rights reserved.
2	#	3	#
3	# Redistribution and use in source and binary forms, with or without	4	# Redistribution and use in source and binary forms, with or without
4	# modification, are permitted provided that the following conditions are	5	# modification, are permitted provided that the following conditions are
Lines 29-34 a/Tools/Scripts/webkitpy/tool/commands/earlywarningsystem_unittest.py_sec2
29	from webkitpy.thirdparty.mock import Mock	30	from webkitpy.thirdparty.mock import Mock
30	from webkitpy.common.host import Host	31	from webkitpy.common.host import Host
31	from webkitpy.common.host_mock import MockHost	32	from webkitpy.common.host_mock import MockHost
		33	from webkitpy.common.net.jsctestresults import JSCTestResults
32	from webkitpy.common.net.layouttestresults import LayoutTestResults	34	from webkitpy.common.net.layouttestresults import LayoutTestResults
33	from webkitpy.common.system.outputcapture import OutputCapture	35	from webkitpy.common.system.outputcapture import OutputCapture
34	from webkitpy.layout_tests.models import test_results	36	from webkitpy.layout_tests.models import test_results
Lines 47-52 class AbstractEarlyWarningSystemTest(QueuesTest): a/Tools/Scripts/webkitpy/tool/commands/earlywarningsystem_unittest.py_sec3
47	class TestEWS(AbstractEarlyWarningSystem):	49	class TestEWS(AbstractEarlyWarningSystem):
48	port_name = "win" # Needs to be a port which port/factory understands.	50	port_name = "win" # Needs to be a port which port/factory understands.
49	_build_style = None	51	_build_style = None
		52	_group = None
50		53
51	ews = TestEWS()	54	ews = TestEWS()
52	ews.bind_to_tool(MockTool())	55	ews.bind_to_tool(MockTool())
Lines 60-65 class AbstractEarlyWarningSystemTest(QueuesTest): a/Tools/Scripts/webkitpy/tool/commands/earlywarningsystem_unittest.py_sec4
60	patch = ews._tool.bugs.fetch_attachment(10000)	63	patch = ews._tool.bugs.fetch_attachment(10000)
61	self.assertMultiLineEqual(ews._failing_tests_message(task, patch), "New failing tests:\nfoo.html\nbar.html")	64	self.assertMultiLineEqual(ews._failing_tests_message(task, patch), "New failing tests:\nfoo.html\nbar.html")
62		65
		66	def test_failing_jsc_tests_message(self):
		67	# Needed to define port_name, used in AbstractEarlyWarningSystem.__init__
		68	class TestEWS(AbstractEarlyWarningSystem):
		69	port_name = "win" # Needs to be a port which port/factory understands.
		70	_build_style = None
		71	_group = "jsc"
		72
		73	ews = TestEWS()
		74	ews.bind_to_tool(MockTool())
		75	ews.host = MockHost()
		76	ews._options = MockOptions(port=None, confirm=False)
		77	OutputCapture().assert_outputs(self, ews.begin_work_queue, expected_logs=self._default_begin_work_queue_logs(ews.name))
		78	task = Mock()
		79	task.results_from_patch_test_run = lambda a: JSCTestResults(False, ["es6.yaml/es6/typed_arrays_Int8Array.js.default", "es6.yaml/es6/typed_arrays_Uint8Array.js.default"])
		80	patch = ews._tool.bugs.fetch_attachment(10000)
		81	self.assertMultiLineEqual(ews._failing_tests_message(task, patch), "New failing tests:\nes6.yaml/es6/typed_arrays_Int8Array.js.default\nes6.yaml/es6/typed_arrays_Uint8Array.js.default\napiTests")
		82
63		83
64	class EarlyWarningSystemTest(QueuesTest):	84	class EarlyWarningSystemTest(QueuesTest):
65	def _default_expected_logs(self, ews):	85	def _default_expected_logs(self, ews):
Lines 68-76 class EarlyWarningSystemTest(QueuesTest): a/Tools/Scripts/webkitpy/tool/commands/earlywarningsystem_unittest.py_sec5
68	"port": ews.port_name,	88	"port": ews.port_name,
69	"architecture": " --architecture=%s" % ews.architecture if ews.architecture else "",	89	"architecture": " --architecture=%s" % ews.architecture if ews.architecture else "",
70	"build_style": ews.build_style(),	90	"build_style": ews.build_style(),
		91	"group": ews.group(),
71	}	92	}
72	if ews.run_tests:	93	if ews.run_tests:
73	run_tests_line = "Running: webkit-patch --status-host=example.com build-and-test --no-clean --no-update --test --non-interactive --build-style=%(build_style)s --port=%(port)s%(architecture)s\n" % string_replacements	94	run_tests_line = "Running: webkit-patch --status-host=example.com build-and-test --no-clean --no-update --test --non-interactive --build-style=%(build_style)s --group=%(group)s --port=%(port)s%(architecture)s\n" % string_replacements
74	else:	95	else:
75	run_tests_line = ""	96	run_tests_line = ""
76	string_replacements['run_tests_line'] = run_tests_line	97	string_replacements['run_tests_line'] = run_tests_line
Lines 81-87 class EarlyWarningSystemTest(QueuesTest): a/Tools/Scripts/webkitpy/tool/commands/earlywarningsystem_unittest.py_sec6
81	Running: webkit-patch --status-host=example.com clean --port=%(port)s%(architecture)s	102	Running: webkit-patch --status-host=example.com clean --port=%(port)s%(architecture)s
82	Running: webkit-patch --status-host=example.com update --port=%(port)s%(architecture)s	103	Running: webkit-patch --status-host=example.com update --port=%(port)s%(architecture)s
83	Running: webkit-patch --status-host=example.com apply-attachment --no-update --non-interactive 10000 --port=%(port)s%(architecture)s	104	Running: webkit-patch --status-host=example.com apply-attachment --no-update --non-interactive 10000 --port=%(port)s%(architecture)s
84	Running: webkit-patch --status-host=example.com build --no-clean --no-update --build-style=%(build_style)s --port=%(port)s%(architecture)s	105	Running: webkit-patch --status-host=example.com check-patch-relevance --group=%(group)s --port=%(port)s%(architecture)s
		106	Running: webkit-patch --status-host=example.com build --no-clean --no-update --build-style=%(build_style)s --group=%(group)s --port=%(port)s%(architecture)s
85	%(run_tests_line)sMOCK: update_status: %(name)s Pass	107	%(run_tests_line)sMOCK: update_status: %(name)s Pass
86	MOCK: release_work_item: %(name)s 10000	108	MOCK: release_work_item: %(name)s 10000
87	""" % string_replacements,	109	""" % string_replacements,



# Copyright (C) 2010 Google Inc. All rights reserved.
# Copyright (C) 2017 Apple Inc. All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are

from webkitpy.tool.steps.applywatchlist import ApplyWatchList
from webkitpy.tool.steps.attachtobug import AttachToBug
from webkitpy.tool.steps.build import Build
from webkitpy.tool.steps.checkpatchrelevance import CheckPatchRelevance
from webkitpy.tool.steps.checkstyle import CheckStyle
from webkitpy.tool.steps.cleanworkingdirectory import CleanWorkingDirectory
from webkitpy.tool.steps.closebug import CloseBug



# Copyright (C) 2010 Google Inc. All rights reserved.
# Copyright (C) 2017 Apple Inc. All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are

            Options.build,
            Options.quiet,
            Options.build_style,
            Options.group,
        ]

    def build(self, build_style, group):
        environment = self._tool.copy_current_environment()
        environment.disable_gcc_smartquotes()
        environment.disable_jhbuild_VT100_output()
        env = environment.to_dictionary()

        if group == "jsc":
            build_command = self._tool.deprecated_port().build_jsc_command(build_style=build_style)
        else:
            build_command = self._tool.deprecated_port().build_webkit_command(build_style=build_style)

        if self._options.architecture:
            build_command += ['ARCHS=%s' % self._options.architecture]

        self._tool.executive.run_and_throw_if_fail(build_command, self._options.quiet,
            cwd=self._tool.scm().checkout_root, env=env)

    def run(self, state):
        if not self._options.build:
            return
        _log.info("Building WebKit")

        group = None
        if hasattr(self._options, 'group'):
            group = self._options.group

        if self._options.build_style == "both":
            self.build("debug", group)
            self.build("release", group)
        else:
            self.build(self._options.build_style, group)



# Copyright (C) 2017 Apple Inc. All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions
# are met:
# 1.  Redistributions of source code must retain the above copyright
#     notice, this list of conditions and the following disclaimer.
# 2.  Redistributions in binary form must reproduce the above copyright
#     notice, this list of conditions and the following disclaimer in the
#     documentation and/or other materials provided with the distribution.
#
# THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS'' AND
# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
# DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS BE LIABLE FOR
# ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
# SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

import logging
import re

from webkitpy.common.system.executive import ScriptError
from webkitpy.tool.steps.abstractstep import AbstractStep
from webkitpy.tool.steps.options import Options

_log = logging.getLogger(__name__)


class CheckPatchRelevance(AbstractStep):
    @classmethod
    def options(cls):
        return AbstractStep.options() + [
            Options.group,
        ]

    jsc_paths = [
        "JSTests/",
        "Source/JavaScriptCore/",
        "Source/WTF/"
        "Source/bmalloc/",
    ]

    def _does_contain_change_in_paths(self, files, patterns):
        for item in files:
            for pattern in patterns:
                if re.search(pattern, item, re.IGNORECASE):
                    return True

        return False

    def run(self, state):
        _log.info("Checking relevance of patch")

        group = None
        if hasattr(self._options, 'group'):
            group = self._options.group

        change_list = self._tool.scm().changed_files()

        if group == "jsc":
            relevant = self._does_contain_change_in_paths(change_list, self.jsc_paths)
        else:
            relevant = True

        if relevant is True:
            _log.info("This patch has relevant changes.")
            return True
        else:
            _log.info("This patch does not have relevant changes.")
            raise ScriptError(message="This patch does not have relevant changes.")



# Copyright (C) 2010 Google Inc. All rights reserved.
# Copyright (C) 2017 Apple Inc. All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are

    email = make_option("--email", action="store", type="string", dest="email", help="Email address to use in ChangeLogs.")
    force_clean = make_option("--force-clean", action="store_true", dest="force_clean", default=False, help="Clean working directory before applying patches (removes local changes and commits)")
    git_commit = make_option("-g", "--git-commit", action="store", dest="git_commit", help="Operate on a local commit. If a range, the commits are squashed into one. <ref>.... includes the working copy changes. UPSTREAM can be used for the upstream/tracking branch.")
    group = make_option("--group", action="store", dest="group", default=None, help="")
    local_commit = make_option("--local-commit", action="store_true", dest="local_commit", default=False, help="Make a local commit for each applied patch")
    non_interactive = make_option("--non-interactive", action="store_true", dest="non_interactive", default=False, help="Never prompt the user, fail as fast as possible.")
    obsolete_patches = make_option("--no-obsolete", action="store_false", dest="obsolete_patches", default=True, help="Do not obsolete old patches before posting this one.")

Lines 1-4 a/Tools/Scripts/webkitpy/tool/steps/runtests.py_sec1
- a/Tools/Scripts/webkitpy/tool/steps/runtests.py +10 lines
1	# Copyright (C) 2010 Google Inc. All rights reserved.	1	# Copyright (C) 2010 Google Inc. All rights reserved.
		2	# Copyright (C) 2017 Apple Inc. All rights reserved.
2	#	3	#
3	# Redistribution and use in source and binary forms, with or without	4	# Redistribution and use in source and binary forms, with or without
4	# modification, are permitted provided that the following conditions are	5	# modification, are permitted provided that the following conditions are
Lines 53-58 class RunTests(AbstractStep): a/Tools/Scripts/webkitpy/tool/steps/runtests.py_sec2
53	Options.iterate_on_new_tests,	54	Options.iterate_on_new_tests,
54	Options.non_interactive,	55	Options.non_interactive,
55	Options.quiet,	56	Options.quiet,
		57	Options.group,
56	]	58	]
57		59
58	def run(self, state):	60	def run(self, state):
Lines 63-68 class RunTests(AbstractStep): a/Tools/Scripts/webkitpy/tool/steps/runtests.py_sec3
63	if not self._options.test:	65	if not self._options.test:
64	return	66	return
65		67
		68	if hasattr(self._options, "group") and self._options.group == "jsc":
		69	self._run_javascriptcore_tests()
		70	return
		71
66	python_unittests_command = self._tool.deprecated_port().run_python_unittests_command()	72	python_unittests_command = self._tool.deprecated_port().run_python_unittests_command()
67	if python_unittests_command:	73	if python_unittests_command:
68	_log.info("Running Python unit tests")	74	_log.info("Running Python unit tests")
Lines 143-145 class RunTests(AbstractStep): a/Tools/Scripts/webkitpy/tool/steps/runtests.py_sec4
143	args.append("--iterations=%d" % iterate_on_new_tests)	149	args.append("--iterations=%d" % iterate_on_new_tests)
144		150
145	self._tool.executive.run_and_throw_if_fail(args, cwd=self._tool.scm().checkout_root)	151	self._tool.executive.run_and_throw_if_fail(args, cwd=self._tool.scm().checkout_root)
		152
		153	def _run_javascriptcore_tests(self):
		154	args = self._tool.deprecated_port().run_javascriptcore_tests_command(self._options.build_style)
		155	self._tool.executive.run_and_throw_if_fail(args, cwd=self._tool.scm().checkout_root)



# Copyright (C) 2010 Google Inc. All rights reserved.
# Copyright (C) 2017 Apple Inc. All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are


import unittest

from webkitpy.common.system.executive import ScriptError
from webkitpy.common.system.outputcapture import OutputCapture
from webkitpy.common.config.ports import DeprecatedPort
from webkitpy.tool.mocktool import MockOptions, MockTool

Running Perl unit tests
MOCK run_and_throw_if_fail: ['Tools/Scripts/test-webkitperl'], cwd=/mock-checkout
Running JavaScriptCore tests
MOCK run_and_throw_if_fail: ['Tools/Scripts/run-javascriptcore-tests', '--no-fail-fast', '--json-output=jsc_test_results.json'], cwd=/mock-checkout
Running bindings generation tests
MOCK run_and_throw_if_fail: ['Tools/Scripts/run-bindings-tests'], cwd=/mock-checkout
Running run-webkit-tests

Running Perl unit tests
MOCK run_and_throw_if_fail: ['Tools/Scripts/test-webkitperl'], cwd=/mock-checkout
Running JavaScriptCore tests
MOCK run_and_throw_if_fail: ['Tools/Scripts/run-javascriptcore-tests', '--no-fail-fast', '--json-output=jsc_test_results.json'], cwd=/mock-checkout
Running bindings generation tests
MOCK run_and_throw_if_fail: ['Tools/Scripts/run-bindings-tests'], cwd=/mock-checkout
Running run-webkit-tests
MOCK run_and_throw_if_fail: ['Tools/Scripts/run-webkit-tests', '--debug', '--quiet'], cwd=/mock-checkout
"""
        OutputCapture().assert_outputs(self, step.run, [{}], expected_logs=expected_logs)

    def test_runtests_jsc(self):
        mock_options = self._step_options()
        mock_options.non_interactive = False
        mock_options.build_style = "release"
        mock_options.group = "jsc"
        step = steps.RunTests(MockTool(log_executive=True), mock_options)
        tool = MockTool(log_executive=True)
        # FIXME: We shouldn't use a real port-object here, but there is too much to mock at the moment.
        tool._deprecated_port = DeprecatedPort()
        step = steps.RunTests(tool, mock_options)
        expected_logs = """MOCK run_and_throw_if_fail: ['Tools/Scripts/run-javascriptcore-tests', '--no-fail-fast', '--json-output=jsc_test_results.json'], cwd=/mock-checkout
"""
        OutputCapture().assert_outputs(self, step.run, [{}], expected_logs=expected_logs)

    def test_runtests_jsc_debug(self):
        mock_options = self._step_options()
        mock_options.non_interactive = False
        mock_options.build_style = "debug"
        mock_options.group = "jsc"
        tool = MockTool(log_executive=True)
        # FIXME: We shouldn't use a real port-object here, but there is too much to mock at the moment.
        tool._deprecated_port = DeprecatedPort()
        step = steps.RunTests(tool, mock_options)
        expected_logs = """MOCK run_and_throw_if_fail: ['Tools/Scripts/run-javascriptcore-tests', '--no-fail-fast', '--json-output=jsc_test_results.json', '--debug'], cwd=/mock-checkout
"""
        OutputCapture().assert_outputs(self, step.run, [{}], expected_logs=expected_logs)

    def test_build_jsc_debug(self):
        mock_options = self._step_options()
        mock_options.non_interactive = False
        mock_options.build_style = "debug"
        mock_options.build = True
        mock_options.architecture = True
        mock_options.group = "jsc"
        tool = MockTool(log_executive=True)
        # FIXME: We shouldn't use a real port-object here, but there is too much to mock at the moment.
        tool._deprecated_port = DeprecatedPort()
        step = steps.Build(tool, mock_options)
        expected_logs = """Building WebKit
MOCK run_and_throw_if_fail: ['Tools/Scripts/build-jsc', '--debug', 'ARCHS=True'], cwd=/mock-checkout, env={'LC_ALL': 'C', 'TERM': 'none', 'MOCK_ENVIRON_COPY': '1'}
"""
        OutputCapture().assert_outputs(self, step.run, [{}], expected_logs=expected_logs)

    def test_build_jsc(self):
        mock_options = self._step_options()
        mock_options.non_interactive = False
        mock_options.build_style = "release"
        mock_options.build = True
        mock_options.architecture = True
        mock_options.group = "jsc"
        tool = MockTool(log_executive=True)
        # FIXME: We shouldn't use a real port-object here, but there is too much to mock at the moment.
        tool._deprecated_port = DeprecatedPort()
        step = steps.Build(tool, mock_options)
        expected_logs = """Building WebKit
MOCK run_and_throw_if_fail: ['Tools/Scripts/build-jsc', '--release', 'ARCHS=True'], cwd=/mock-checkout, env={'LC_ALL': 'C', 'TERM': 'none', 'MOCK_ENVIRON_COPY': '1'}
"""
        OutputCapture().assert_outputs(self, step.run, [{}], expected_logs=expected_logs)

    def test_patch_relevant(self):
        self.maxDiff = None
        mock_options = self._step_options()
        tool = MockTool(log_executive=True)
        tool.scm()._mockChangedFiles = ["JSTests/MockFile1", "ChangeLog"]
        # FIXME: We shouldn't use a real port-object here, but there is too much to mock at the moment.
        tool._deprecated_port = DeprecatedPort()
        step = steps.CheckPatchRelevance(tool, mock_options)
        expected_logs = """Checking relevance of patch
This patch has relevant changes.
"""
        OutputCapture().assert_outputs(self, step.run, [{}], expected_logs=expected_logs)

    def test_patch_relevant_jsc(self):
        self.maxDiff = None
        mock_options = self._step_options()
        mock_options.group = "jsc"
        tool = MockTool(log_executive=True)
        tool.scm()._mockChangedFiles = ["JSTests/MockFile1", "ChangeLog"]
        # FIXME: We shouldn't use a real port-object here, but there is too much to mock at the moment.
        tool._deprecated_port = DeprecatedPort()
        step = steps.CheckPatchRelevance(tool, mock_options)
        expected_logs = """Checking relevance of patch
This patch has relevant changes.
"""
        OutputCapture().assert_outputs(self, step.run, [{}], expected_logs=expected_logs)

    def test_patch_not_relevant_jsc(self):
        self.maxDiff = None
        mock_options = self._step_options()
        mock_options.group = "jsc"
        tool = MockTool(log_executive=True)
        tool.scm()._mockChangedFiles = ["Tools/ChangeLog", "Tools/Scripts/webkitpy/tool/steps/steps_unittest.py"]
        # FIXME: We shouldn't use a real port-object here, but there is too much to mock at the moment.
        tool._deprecated_port = DeprecatedPort()
        step = steps.CheckPatchRelevance(tool, mock_options)
        expected_logs = """Checking relevance of patch
This patch does not have relevant changes.
"""
        with self.assertRaises(ScriptError):
            OutputCapture().assert_outputs(self, step.run, [{}], expected_logs=expected_logs)

Return to Bug 162458

Lines 1-3 a/Tools/ChangeLog_sec1
- a/Tools/ChangeLog +114 lines
		1	2017-01-30 Srinivasan Vijayaraghavan <svijayaraghavan@apple.com>
		2
		3	EWS should run JavaScriptCore tests
		4	https://bugs.webkit.org/show_bug.cgi?id=162458
		5
		6	Reviewed by NOBODY (OOPS!).
		7
		8	* Scripts/webkitpy/common/checkout/scm/scm_mock.py:
		9	(MockSCM.__init__): Add _mockChangedFiles variable.
		10	(MockSCM.changed_files): Change to use _mockChangedFiles variables.
		11	* Scripts/webkitpy/common/config/ews.json: Add config info for JSC EWS.
		12	* Scripts/webkitpy/common/config/ports.py: Add support for JSC EWS in Mac Port.
		13	(DeprecatedPort.build_jsc_command): Added - command to build JSC only (quicker than building all of WebKit).
		14	(DeprecatedPort.run_javascriptcore_tests_command): Allow JSC EWS to only run JSC tests.
		15	(MacPort.run_webkit_tests_command): Check for JSC.
		16	* Scripts/webkitpy/common/config/ports_mock.py:
		17	(MockPort.run_javascriptcore_tests_command): Add build_style argument.
		18	* Scripts/webkitpy/common/config/ports_unittest.py:
		19	(DeprecatedPortTest.test_mac_port): Add unit tests for build-jsc shell command.
		20	* Scripts/webkitpy/common/net/jsctestresults.py: Added.
		21	(JSCTestResults): Added.
		22	(JSCTestResults.intersection): Return a JSCTestResults object with failures common to both input objects.
		23	(JSCTestResults.results_from_string): Use json library to parse results.
		24	(JSCTestResults.__init__): Initialize members.
		25	(JSCTestResults.equals): This enables unit testing.
		26	(JSCTestResults.is_subset): Checks if one set of failures is a subset of another.
		27	(JSCTestResults.stress_test_failures): Getter.
		28	(JSCTestResults.all_api_tests_passed): Getter.
		29	(JSCTestResults.all_passed): Getter.
		30	(JSCTestResults.failing_test_results): Getter. Some of these exist for interopability reasons.
		31	(JSCTestResults.test_results): Getter.
		32	(JSCTestResults.failing_tests): Getter.
		33	(JSCTestResults.did_exceed_test_failure_limit): Getter.
		34	* Scripts/webkitpy/common/net/jsctestresults_unittest.py: Added.
		35	(JSCTestResultsTest): Class to test JSCTestResults.
		36	(JSCTestResultsTest.test_results_from_string): Tests parsing.
		37	(JSCTestResultsTest.test_intersection_api_tests): Unit test for intersection() class method.
		38	(JSCTestResultsTest.test_intersection_stress_tests): Unit test for intersection() class method.
		39	(JSCTestResultsTest.test_intersection_general_case): Unit test for intersection() class method.
		40	* Scripts/webkitpy/tool/bot/earlywarningsystemtask.py:
		41	(EarlyWarningSystemTask.run): Add check_patch_relevance step.
		42	* Scripts/webkitpy/tool/bot/jscews_unittest.py: Added.
		43	(MockPatchAnalysisTask): Mocked-out version of PatchAnalysisTask that doesn't run commands.
		44	(MockPatchAnalysisTask.__init__): Sets attributes.
		45	(MockPatchAnalysisTask._test): Override to not run command.
		46	(MockPatchAnalysisTask._build_and_test_without_patch): Override to not run command.
		47	(MockPatchAnalysisTask.validate): Assume mocked patch is valid for purposes of testing retry logic.
		48	(MockPatchAnalysisTask.test_run_count): Specific to the mocked version, to test retry logic.
		49	(MockJSCEarlyWarningSystem): Mocked-out version of AbstarctEarlyWarningSystem so we can provide test results.
		50	(MockJSCEarlyWarningSystem.__init__): Sets attributes, also sets group to jsc.
		51	(MockJSCEarlyWarningSystem.test_results): Returns test results provided by us, instead of using a JSON reader.
		52	(JSCEarlyWarningSystemTest): Class to test retry logic in below situations.
		53	(JSCEarlyWarningSystemTest._create_task): Helper function to abstract out common code.
		54	(JSCEarlyWarningSystemTest._results_indicate_all_passed): False if input is None or has failures, else True.
		55	(JSCEarlyWarningSystemTest.test_success_case): Clean patch on clean tree.
		56	(JSCEarlyWarningSystemTest.test_test_failure): Failed patch on clean tree.
		57	(JSCEarlyWarningSystemTest.test_fix): Patch that fixes all tree redness.
		58	(JSCEarlyWarningSystemTest.test_ineffective_patch): Patch that has same failures as tree.
		59	(JSCEarlyWarningSystemTest.test_partially_effective_patch): Patch fixes some failures but adds no new failures.
		60	(JSCEarlyWarningSystemTest.test_different_test_failures_in_patch_and_tree): Patch has some failures not in tree.
		61	(JSCEarlyWarningSystemTest.test_first_results_could_not_be_read): Patch results not readable.
		62	(JSCEarlyWarningSystemTest.test_second_results_could_not_be_read): Patch results not readable on second run.
		63	(JSCEarlyWarningSystemTest.test_clean_results_could_not_be_read): Results from clean tree not readable.
		64	(JSCEarlyWarningSystemTest.test_flaky_results_on_clean_tree_pass): Patch has one flake and no failures.
65	(JSCEarlyWarningSystemTest.test_flaky_results_on_clean_tree_pass_v2): Patch has one flake and no failures.
66	(JSCEarlyWarningSystemTest.test_flaky_results_on_clean_tree_failure): Patch has flakes and failed tests.
67	(JSCEarlyWarningSystemTest.test_flaky_results_on_red_tree_pass): Patch has same failures as tree, plus a flake.
68	* Scripts/webkitpy/tool/bot/jsctestresultsreader.py: Added.
69	(JSCTestResultsReader): Reads results file.
70	(JSCTestResultsReader.__init__): Sets attributes.
71	(JSCTestResultsReader._read_file_contents): Reads file.
72	(JSCTestResultsReader._create_jsc_test_results): Creates a JSCTestResults object.
73	(JSCTestResultsReader.results): Invokes _create_jsc_test_results to return a JSCTestResults object.
74	* Scripts/webkitpy/tool/bot/patchanalysistask.py:
75	(PatchIsNotApplicable): Exception for when patch doesn't have relevant changes.
76	(PatchAnalysisTask._check_patch_relevance): Added.
77	(PatchAnalysisTask._build): Check for JSC.
78	(PatchAnalysisTask._build_without_patch): Check for JSC.
79	(PatchAnalysisTask._test): Check for JSC.
80	(PatchAnalysisTask._build_and_test_without_patch): Check for JSC.
81	(PatchAnalysisTask._test_patch): Add retry logic for JSC.
82	* Scripts/webkitpy/tool/commands/download.py:
83	(CheckPatchRelevance): Add check-patch-relevance command.
84	* Scripts/webkitpy/tool/commands/earlywarningsystem.py:
85	(AbstractEarlyWarningSystem.begin_work_queue): Use JSCTestResultsReader not LayoutTestResultsReader in JSC EWS.
86	(AbstractEarlyWarningSystem.review_patch): Handle PatchIsApplicable
87	(AbstractEarlyWarningSystem.test_results): _layout_test_results_reader -> _test_results_reader.
88	(AbstractEarlyWarningSystem.archive_last_test_results): _layout_test_results_reader -> _test_results_reader.
89	(AbstractEarlyWarningSystem.group): This attribute determines the type of EWS (eg. JSC).
90	(AbstractEarlyWarningSystem.load_ews_classes): Add _group.
91	* Scripts/webkitpy/tool/commands/earlywarningsystem_unittest.py:
92	(AbstractEarlyWarningSystemTest.test_failing_tests_message.TestEWS): Add _group.
93	(AbstractEarlyWarningSystemTest.test_failing_jsc_tests_message.TestEWS): Added test for jsc failures message.
94	(AbstractEarlyWarningSystemTest): Add _group variable.
95	(EarlyWarningSystemTest._default_expected_logs): Add check-patch-relevance step and --group option.
96	* Scripts/webkitpy/tool/steps/__init__.py: Add CheckPatchRelevance import.
97	* Scripts/webkitpy/tool/steps/build.py:
98	(Build.options): Check for JSC.
99	(Build.build): Check for JSC.
100	(Build.run): Check for JSC.
101	* Scripts/webkitpy/tool/steps/checkpatchrelevance.py: Added.
102	(CheckPatchRelevance): Added.
103	(CheckPatchRelevance._does_contain_change_in_paths): Abstract function to perform regex matching.
104	(CheckPatchRelevance.run): Check if changed files in patch belong in certain folders.
105	* Scripts/webkitpy/tool/steps/options.py:
106	(Options): Add --group command line option.
107	* Scripts/webkitpy/tool/steps/runtests.py:
108	(RunTests.options): Add group.
109	(RunTests.run): Check for JSC.
110	(RunTests._run_webkit_tests): Check for JSC.
111	(RunTests): Add _group attribute.
112	(RunTests._run_javascriptcore_tests): New.
113	* Scripts/webkitpy/tool/steps/steps_unittest.py: Unit tests.
114
1	2016-09-25 Gyuyoung Kim <gyuyoung.kim@webkit.org>	115	2016-09-25 Gyuyoung Kim <gyuyoung.kim@webkit.org>
2		116
3	[EFL] Adjust EFL indentation to EFL MiniBrowser	117	[EFL] Adjust EFL indentation to EFL MiniBrowser

Lines 1-4 a/Tools/Scripts/webkitpy/common/checkout/scm/scm_mock.py_sec1
- a/Tools/Scripts/webkitpy/common/checkout/scm/scm_mock.py -1 / +3 lines
1	# Copyright (C) 2011 Google Inc. All rights reserved.	1	# Copyright (C) 2011 Google Inc. All rights reserved.
		2	# Copyright (C) 2017 Apple Inc. All rights reserved.
2	#	3	#
3	# Redistribution and use in source and binary forms, with or without	4	# Redistribution and use in source and binary forms, with or without
4	# modification, are permitted provided that the following conditions are	5	# modification, are permitted provided that the following conditions are
Lines 37-42 class MockSCM(object): a/Tools/Scripts/webkitpy/common/checkout/scm/scm_mock.py_sec2
37	self.added_paths = set()	38	self.added_paths = set()
38	self._filesystem = filesystem or MockFileSystem()	39	self._filesystem = filesystem or MockFileSystem()
39	self._executive = executive or MockExecutive()	40	self._executive = executive or MockExecutive()
		41	self._mockChangedFiles = ["MockFile1"]
40		42
41	def add(self, destination_path):	43	def add(self, destination_path):
42	self.add_list([destination_path])	44	self.add_list([destination_path])
Lines 71-77 class MockSCM(object): a/Tools/Scripts/webkitpy/common/checkout/scm/scm_mock.py_sec3
71	return self._filesystem.join(self.checkout_root, *comps)	73	return self._filesystem.join(self.checkout_root, *comps)
72		74
73	def changed_files(self, git_commit=None):	75	def changed_files(self, git_commit=None):
74	return ["MockFile1"]	76	return self._mockChangedFiles
75		77
76	def changed_files_for_revision(self, revision):	78	def changed_files_for_revision(self, revision):
77	return ["MockFile1"]	79	return ["MockFile1"]

Line 0 a/Tools/Scripts/webkitpy/common/net/jsctestresults.py_sec1
- a/Tools/Scripts/webkitpy/common/net/jsctestresults.py +86 lines
		1	# Copyright (C) 2017 Apple Inc. All rights reserved.
		2	#
		3	# Redistribution and use in source and binary forms, with or without
		4	# modification, are permitted provided that the following conditions
		5	# are met:
		6	# 1. Redistributions of source code must retain the above copyright
		7	# notice, this list of conditions and the following disclaimer.
		8	# 2. Redistributions in binary form must reproduce the above copyright
		9	# notice, this list of conditions and the following disclaimer in the
		10	# documentation and/or other materials provided with the distribution.
		11	#
		12	# THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS'' AND
		13	# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
		14	# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
		15	# DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS BE LIABLE FOR
		16	# ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
		17	# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
		18	# SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
		19	# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
		20	# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
		21	# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
		22
		23	import json
		24	import logging
		25
		26	_log = logging.getLogger(__name__)
		27
		28
		29	class JSCTestResults(object):
		30	@classmethod
		31	def intersection(cls, first, second):
		32	intersection_api_tests_passed = first._all_api_tests_passed or second._all_api_tests_passed
		33	intersection_stress_test_failures = list(set(first._stress_test_failures) & set(second._stress_test_failures))
		34	return cls(intersection_api_tests_passed, intersection_stress_test_failures)
		35
		36	@classmethod
		37	def results_from_string(cls, string):
		38	if not string:
		39	return None
		40
		41	try:
		42	parsed_results = json.loads(string)
		43	except ValueError:
		44	return None
		45
		46	if 'allApiTestsPassed' not in parsed_results or 'stressTestFailures' not in parsed_results:
		47	return None
		48
		49	return cls(parsed_results['allApiTestsPassed'], parsed_results['stressTestFailures'])
		50
		51	def __init__(self, all_api_tests_passed, stress_test_failures):
		52	self._all_api_tests_passed = all_api_tests_passed
		53	self._stress_test_failures = stress_test_failures
		54
		55	self._test_results = stress_test_failures[:]
		56	if not self._all_api_tests_passed:
		57	self._test_results.append("apiTests")
		58
		59	def equals(self, other):
		60	return self._all_api_tests_passed == other._all_api_tests_passed and \
		61	set(self._stress_test_failures) == set(other._stress_test_failures)
		62
		63	def is_subset(self, other):
		64	return set(self.test_results()) <= set(other.test_results())
65
66	def stress_test_failures(self):
67	return self._stress_test_failures
68
69	def all_api_tests_passed(self):
70	return self._all_api_tests_passed
71
72	def all_passed(self):
73	return self._all_api_tests_passed and not self._stress_test_failures
74
75	def failing_test_results(self):
76	return self.stress_test_failures()
77
78	def test_results(self):
79	return self._test_results
80
81	def failing_tests(self):
82	return self._test_results
83
84	# No defined failure limit for JSC.
85	def did_exceed_test_failure_limit(self):
86	return False

Line 0 a/Tools/Scripts/webkitpy/tool/bot/jscews_unittest.py_sec1
- a/Tools/Scripts/webkitpy/tool/bot/jscews_unittest.py +219 lines
		1	# Copyright (C) 2017 Apple Inc. All rights reserved.
		2	#
		3	# Redistribution and use in source and binary forms, with or without
		4	# modification, are permitted provided that the following conditions
		5	# are met:
		6	# 1. Redistributions of source code must retain the above copyright
		7	# notice, this list of conditions and the following disclaimer.
		8	# 2. Redistributions in binary form must reproduce the above copyright
		9	# notice, this list of conditions and the following disclaimer in the
		10	# documentation and/or other materials provided with the distribution.
		11	#
		12	# THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS'' AND
		13	# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
		14	# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
		15	# DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS BE LIABLE FOR
		16	# ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
		17	# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
		18	# SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
		19	# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
		20	# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
		21	# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
		22
		23	import logging
		24	import unittest
		25
		26	from webkitpy.common.net.jsctestresults import JSCTestResults
		27	from webkitpy.common.system.executive import ScriptError
		28	from webkitpy.tool.bot.patchanalysistask import *
		29	from webkitpy.tool.commands.earlywarningsystem import AbstractEarlyWarningSystem
		30	from webkitpy.tool.mocktool import MockTool
		31
		32	_log = logging.getLogger(__name__)
		33
		34
		35	class MockPatchAnalysisTask(PatchAnalysisTask):
		36	def __init__(self, delegate, patch, patches_passed_all_tests):
		37	self._delegate = delegate
		38	self._patch = patch
		39	self._script_error = None
		40	self._results_archive_from_patch_test_run = None
		41	self._results_from_patch_test_run = None
		42	self.error = None
		43	self._patches_passed_all_tests = patches_passed_all_tests
		44	self._test_run_count = 0
		45	self.failure_status_id = 42
		46
		47	def _test(self):
		48	self._test_run_count += 1
		49
		50	if self._patches_passed_all_tests.pop() == True:
		51	return True
		52	else:
		53	self._script_error = ScriptError("Regression test")
		54	return False
		55
		56	def _build_and_test_without_patch(self):
		57	self._test_run_count += 1
		58	return True
		59
		60	def validate(self):
		61	return True
		62
		63	def test_run_count(self):
		64	return self._test_run_count
65
66
67	# This is the delegate to be used with MockPatchAnalysisTask, above.
68	class MockJSCEarlyWarningSystem(AbstractEarlyWarningSystem):
69	def __init__(self, first_test_results, second_test_results, clean_test_results):
70	AbstractEarlyWarningSystem.__init__(self)
71	self._group = "jsc"
72	self._results_in_order = [clean_test_results, second_test_results, first_test_results]
73
74	def test_results(self):
75	return self._results_in_order.pop()
76
77
78	class JSCEarlyWarningSystemTest(unittest.TestCase):
79	def _results_indicate_all_passed(self, results):
80	if results == None:
81	return False
82	return results.all_passed()
83
84	def _create_task(self, first_test_results, second_test_results, clean_test_results):
85	queue = MockJSCEarlyWarningSystem(first_test_results, second_test_results, clean_test_results)
86	tool = MockTool(log_executive=True)
87	patch = tool.bugs.fetch_attachment(10000)
88	patches_passed_all_tests = map(self._results_indicate_all_passed, [second_test_results, first_test_results])
89	return MockPatchAnalysisTask(queue, patch, patches_passed_all_tests)
90
91	def test_success_case(self):
92	first_test_results = JSCTestResults(True, [])
93	second_test_results = JSCTestResults(True, [])
94	clean_test_results = JSCTestResults(True, [])
95	task = self._create_task(first_test_results, second_test_results, clean_test_results)
96
97	return_value = task._test_patch()
98	self.assertEqual(task.test_run_count(), 1)
99	self.assertEqual(return_value, True)
100
101	def test_test_failure(self):
102	first_test_results = JSCTestResults(True, ['Fail.js'])
103	second_test_results = JSCTestResults(True, ['Fail.js'])
104	clean_test_results = JSCTestResults(True, [])
105	task = self._create_task(first_test_results, second_test_results, clean_test_results)
106
107	with self.assertRaises(ScriptError):
108	return_value = task._test_patch()
109	self.assertEqual(task.test_run_count(), 3)
110
111	def test_fix(self):
112	first_test_results = JSCTestResults(True, [])
113	second_test_results = JSCTestResults(True, [])
114	clean_test_results = JSCTestResults(True, ['Fail.js'])
115	task = self._create_task(first_test_results, second_test_results, clean_test_results)
116
117	return_value = task._test_patch()
118	self.assertEqual(task.test_run_count(), 1)
119	self.assertEqual(return_value, True)
120
121	def test_ineffective_patch(self):
122	first_test_results = JSCTestResults(False, ['failure1.js', 'failure2.js'])
123	second_test_results = JSCTestResults(False, ['failure1.js', 'failure2.js'])
124	clean_test_results = JSCTestResults(False, ['failure1.js', 'failure2.js'])
125	task = self._create_task(first_test_results, second_test_results, clean_test_results)
126
127	return_value = task._test_patch()
128	self.assertEqual(task.test_run_count(), 3)
129	self.assertEqual(return_value, True)
130
131	def test_partially_effective_patch(self):
132	first_test_results = JSCTestResults(True, ['failure2.js'])
133	second_test_results = JSCTestResults(True, ['failure2.js'])
134	clean_test_results = JSCTestResults(False, ['failure1.js', 'failure2.js'])
135	task = self._create_task(first_test_results, second_test_results, clean_test_results)
136
137	return_value = task._test_patch()
138	self.assertEqual(task.test_run_count(), 3)
139	self.assertEqual(return_value, True)
140
141	def test_different_test_failures_in_patch_and_tree(self):
142	first_test_results = JSCTestResults(False, [])
143	second_test_results = JSCTestResults(False, [])
144	clean_test_results = JSCTestResults(True, ['failure1.js', 'failure2.js'])
145	task = self._create_task(first_test_results, second_test_results, clean_test_results)
146
147	with self.assertRaises(ScriptError):
148	return_value = task._test_patch()
149	self.assertEqual(task.test_run_count(), 3)
150
151	def test_first_results_could_not_be_read(self):
152	first_test_results = None
153	second_test_results = JSCTestResults(True, [])
154	clean_test_results = JSCTestResults(True, [])
155	task = self._create_task(first_test_results, second_test_results, clean_test_results)
156
157	with self.assertRaises(ScriptError):
158	return_value = task._test_patch()
159	self.assertEqual(task.test_run_count(), 1)
160
161	def test_second_results_could_not_be_read(self):
162	first_test_results = JSCTestResults(False, ['failure1.js', 'failure2.js'])
163	second_test_results = None
164	clean_test_results = JSCTestResults(True, [])
165	task = self._create_task(first_test_results, second_test_results, clean_test_results)
166
167	with self.assertRaises(ScriptError):
168	return_value = task._test_patch()
169	self.assertEqual(task.test_run_count(), 2)
170
171	def test_clean_results_could_not_be_read(self):
172	first_test_results = JSCTestResults(True, ['failure2.js'])
173	second_test_results = JSCTestResults(True, ['failure2.js'])
174	clean_test_results = None
175	task = self._create_task(first_test_results, second_test_results, clean_test_results)
176
177	with self.assertRaises(ScriptError):
178	return_value = task._test_patch()
179	self.assertEqual(task.test_run_count(), 3)
180
181	def test_flaky_results_on_clean_tree_pass(self):
182	first_test_results = JSCTestResults(True, ['failure2.js'])
183	second_test_results = JSCTestResults(True, [])
184	clean_test_results = JSCTestResults(True, [])
185	task = self._create_task(first_test_results, second_test_results, clean_test_results)
186
187	return_value = task._test_patch()
188	self.assertEqual(return_value, True)
189	self.assertEqual(task.test_run_count(), 2)
190
191	def test_flaky_results_on_clean_tree_pass_v2(self):
192	first_test_results = JSCTestResults(True, [])
193	second_test_results = JSCTestResults(True, ['failure2.js'])
194	clean_test_results = JSCTestResults(True, [])
195	task = self._create_task(first_test_results, second_test_results, clean_test_results)
196
197	return_value = task._test_patch()
198	self.assertEqual(return_value, True)
199	self.assertEqual(task.test_run_count(), 1)
200
201	def test_flaky_results_on_clean_tree_failure(self):
202	first_test_results = JSCTestResults(False, ['failure1.js', 'failure2.js'])
203	second_test_results = JSCTestResults(True, ['failure2.js'])
204	clean_test_results = JSCTestResults(True, [])
205	task = self._create_task(first_test_results, second_test_results, clean_test_results)
206
207	with self.assertRaises(ScriptError):
208	return_value = task._test_patch()
209	self.assertEqual(task.test_run_count(), 3)
210
211	def test_flaky_results_on_red_tree_pass(self):
212	first_test_results = JSCTestResults(True, ['failure1.js'])
213	second_test_results = JSCTestResults(True, ['failure1.js', 'failure2.js'])
214	clean_test_results = JSCTestResults(True, ['failure1.js'])
215	task = self._create_task(first_test_results, second_test_results, clean_test_results)
216
217	return_value = task._test_patch()
218	self.assertEqual(return_value, True)
219	self.assertEqual(task.test_run_count(), 3)

Line 0 a/Tools/Scripts/webkitpy/tool/bot/jsctestresultsreader.py_sec1
- a/Tools/Scripts/webkitpy/tool/bot/jsctestresultsreader.py +47 lines
	1	# Copyright (C) 2017 Apple Inc. All rights reserved.
	2	#
	3	# Redistribution and use in source and binary forms, with or without
	4	# modification, are permitted provided that the following conditions
	5	# are met:
	6	# 1. Redistributions of source code must retain the above copyright
	7	# notice, this list of conditions and the following disclaimer.
	8	# 2. Redistributions in binary form must reproduce the above copyright
	9	# notice, this list of conditions and the following disclaimer in the
	10	# documentation and/or other materials provided with the distribution.
	11	#
	12	# THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS'' AND
	13	# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
	14	# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
	15	# DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS BE LIABLE FOR
	16	# ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
	17	# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
	18	# SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
	19	# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
	20	# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
	21	# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
	22
	23	import logging
	24
	25	from webkitpy.common.net.jsctestresults import JSCTestResults
	26
	27	_log = logging.getLogger(__name__)
	28
	29
	30	class JSCTestResultsReader(object):
	31	def __init__(self, host, results_directory):
	32	self._host = host
	33	self._results_directory = results_directory
	34
	35	def _read_file_contents(self, path):
	36	try:
	37	return self._host.filesystem.read_text_file(path)
	38	except IOError, e: # File does not exist or can't be read.
	39	return None
	40
	41	def _create_jsc_test_results(self):
	42	results_path = self._host.filesystem.join(self._results_directory, "jsc_test_results.json")
	43	contents = self._read_file_contents(results_path)
	44	return JSCTestResults.results_from_string(contents)
	45
	46	def results(self):
	47	return self._create_jsc_test_results()

Lines 1-4 a/Tools/Scripts/webkitpy/tool/steps/__init__.py_sec1
- a/Tools/Scripts/webkitpy/tool/steps/__init__.py +2 lines
1	# Copyright (C) 2010 Google Inc. All rights reserved.	1	# Copyright (C) 2010 Google Inc. All rights reserved.
		2	# Copyright (C) 2017 Apple Inc. All rights reserved.
2	#	3	#
3	# Redistribution and use in source and binary forms, with or without	4	# Redistribution and use in source and binary forms, with or without
4	# modification, are permitted provided that the following conditions are	5	# modification, are permitted provided that the following conditions are
Lines 33-38 from webkitpy.tool.steps.applypatchwithlocalcommit import ApplyPatchWithLocalCom a/Tools/Scripts/webkitpy/tool/steps/__init__.py_sec2
33	from webkitpy.tool.steps.applywatchlist import ApplyWatchList	34	from webkitpy.tool.steps.applywatchlist import ApplyWatchList
34	from webkitpy.tool.steps.attachtobug import AttachToBug	35	from webkitpy.tool.steps.attachtobug import AttachToBug
35	from webkitpy.tool.steps.build import Build	36	from webkitpy.tool.steps.build import Build
		37	from webkitpy.tool.steps.checkpatchrelevance import CheckPatchRelevance
36	from webkitpy.tool.steps.checkstyle import CheckStyle	38	from webkitpy.tool.steps.checkstyle import CheckStyle
37	from webkitpy.tool.steps.cleanworkingdirectory import CleanWorkingDirectory	39	from webkitpy.tool.steps.cleanworkingdirectory import CleanWorkingDirectory
38	from webkitpy.tool.steps.closebug import CloseBug	40	from webkitpy.tool.steps.closebug import CloseBug

Lines 1-4 a/Tools/Scripts/webkitpy/tool/steps/build.py_sec1
- a/Tools/Scripts/webkitpy/tool/steps/build.py -7 / +17 lines
1	# Copyright (C) 2010 Google Inc. All rights reserved.	1	# Copyright (C) 2010 Google Inc. All rights reserved.
		2	# Copyright (C) 2017 Apple Inc. All rights reserved.
2	#	3	#
3	# Redistribution and use in source and binary forms, with or without	4	# Redistribution and use in source and binary forms, with or without
4	# modification, are permitted provided that the following conditions are	5	# modification, are permitted provided that the following conditions are
Lines 41-68 class Build(AbstractStep): a/Tools/Scripts/webkitpy/tool/steps/build.py_sec2
41	Options.build,	42	Options.build,
42	Options.quiet,	43	Options.quiet,
43	Options.build_style,	44	Options.build_style,
		45	Options.group,
44	]	46	]
45		47
46	def build(self, build_style):	48	def build(self, build_style, group):
47	environment = self._tool.copy_current_environment()	49	environment = self._tool.copy_current_environment()
48	environment.disable_gcc_smartquotes()	50	environment.disable_gcc_smartquotes()
49	environment.disable_jhbuild_VT100_output()	51	environment.disable_jhbuild_VT100_output()
50	env = environment.to_dictionary()	52	env = environment.to_dictionary()
51		53
52	build_webkit_command = self._tool.deprecated_port().build_webkit_command(build_style=build_style)	54	if group == "jsc":
		55	build_command = self._tool.deprecated_port().build_jsc_command(build_style=build_style)
		56	else:
		57	build_command = self._tool.deprecated_port().build_webkit_command(build_style=build_style)
53		58
54	if self._options.architecture:	59	if self._options.architecture:
55	build_webkit_command += ['ARCHS=%s' % self._options.architecture]	60	build_command += ['ARCHS=%s' % self._options.architecture]
56		61
57	self._tool.executive.run_and_throw_if_fail(build_webkit_command, self._options.quiet,	62	self._tool.executive.run_and_throw_if_fail(build_command, self._options.quiet,
58	cwd=self._tool.scm().checkout_root, env=env)	63	cwd=self._tool.scm().checkout_root, env=env)
59		64
60	def run(self, state):	65	def run(self, state):
61	if not self._options.build:	66	if not self._options.build:
62	return	67	return
63	_log.info("Building WebKit")	68	_log.info("Building WebKit")
		69
		70	group = None
		71	if hasattr(self._options, 'group'):
		72	group = self._options.group
		73
64	if self._options.build_style == "both":	74	if self._options.build_style == "both":
65	self.build("debug")	75	self.build("debug", group)
66	self.build("release")	76	self.build("release", group)
67	else:	77	else:
68	self.build(self._options.build_style)	78	self.build(self._options.build_style, group)

Line 0 a/Tools/Scripts/webkitpy/tool/steps/checkpatchrelevance.py_sec1
- a/Tools/Scripts/webkitpy/tool/steps/checkpatchrelevance.py +74 lines
		1	# Copyright (C) 2017 Apple Inc. All rights reserved.
		2	#
		3	# Redistribution and use in source and binary forms, with or without
		4	# modification, are permitted provided that the following conditions
		5	# are met:
		6	# 1. Redistributions of source code must retain the above copyright
		7	# notice, this list of conditions and the following disclaimer.
		8	# 2. Redistributions in binary form must reproduce the above copyright
		9	# notice, this list of conditions and the following disclaimer in the
		10	# documentation and/or other materials provided with the distribution.
		11	#
		12	# THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS'' AND
		13	# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
		14	# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
		15	# DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS BE LIABLE FOR
		16	# ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
		17	# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
		18	# SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
		19	# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
		20	# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
		21	# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
		22
		23	import logging
		24	import re
		25
		26	from webkitpy.common.system.executive import ScriptError
		27	from webkitpy.tool.steps.abstractstep import AbstractStep
		28	from webkitpy.tool.steps.options import Options
		29
		30	_log = logging.getLogger(__name__)
		31
		32
		33	class CheckPatchRelevance(AbstractStep):
		34	@classmethod
		35	def options(cls):
		36	return AbstractStep.options() + [
		37	Options.group,
		38	]
		39
		40	jsc_paths = [
		41	"JSTests/",
		42	"Source/JavaScriptCore/",
		43	"Source/WTF/"
		44	"Source/bmalloc/",
		45	]
		46
		47	def _does_contain_change_in_paths(self, files, patterns):
		48	for item in files:
		49	for pattern in patterns:
		50	if re.search(pattern, item, re.IGNORECASE):
		51	return True
		52
		53	return False
		54
		55	def run(self, state):
		56	_log.info("Checking relevance of patch")
		57
		58	group = None
		59	if hasattr(self._options, 'group'):
		60	group = self._options.group
		61
		62	change_list = self._tool.scm().changed_files()
		63
		64	if group == "jsc":
65	relevant = self._does_contain_change_in_paths(change_list, self.jsc_paths)
66	else:
67	relevant = True
68
69	if relevant is True:
70	_log.info("This patch has relevant changes.")
71	return True
72	else:
73	_log.info("This patch does not have relevant changes.")
74	raise ScriptError(message="This patch does not have relevant changes.")

Lines 1-4 a/Tools/Scripts/webkitpy/tool/steps/options.py_sec1
- a/Tools/Scripts/webkitpy/tool/steps/options.py +2 lines
1	# Copyright (C) 2010 Google Inc. All rights reserved.	1	# Copyright (C) 2010 Google Inc. All rights reserved.
		2	# Copyright (C) 2017 Apple Inc. All rights reserved.
2	#	3	#
3	# Redistribution and use in source and binary forms, with or without	4	# Redistribution and use in source and binary forms, with or without
4	# modification, are permitted provided that the following conditions are	5	# modification, are permitted provided that the following conditions are
Lines 46-51 class Options(object): a/Tools/Scripts/webkitpy/tool/steps/options.py_sec2
46	email = make_option("--email", action="store", type="string", dest="email", help="Email address to use in ChangeLogs.")	47	email = make_option("--email", action="store", type="string", dest="email", help="Email address to use in ChangeLogs.")
47	force_clean = make_option("--force-clean", action="store_true", dest="force_clean", default=False, help="Clean working directory before applying patches (removes local changes and commits)")	48	force_clean = make_option("--force-clean", action="store_true", dest="force_clean", default=False, help="Clean working directory before applying patches (removes local changes and commits)")
48	git_commit = make_option("-g", "--git-commit", action="store", dest="git_commit", help="Operate on a local commit. If a range, the commits are squashed into one. <ref>.... includes the working copy changes. UPSTREAM can be used for the upstream/tracking branch.")	49	git_commit = make_option("-g", "--git-commit", action="store", dest="git_commit", help="Operate on a local commit. If a range, the commits are squashed into one. <ref>.... includes the working copy changes. UPSTREAM can be used for the upstream/tracking branch.")
		50	group = make_option("--group", action="store", dest="group", default=None, help="")
49	local_commit = make_option("--local-commit", action="store_true", dest="local_commit", default=False, help="Make a local commit for each applied patch")	51	local_commit = make_option("--local-commit", action="store_true", dest="local_commit", default=False, help="Make a local commit for each applied patch")
50	non_interactive = make_option("--non-interactive", action="store_true", dest="non_interactive", default=False, help="Never prompt the user, fail as fast as possible.")	52	non_interactive = make_option("--non-interactive", action="store_true", dest="non_interactive", default=False, help="Never prompt the user, fail as fast as possible.")
51	obsolete_patches = make_option("--no-obsolete", action="store_false", dest="obsolete_patches", default=True, help="Do not obsolete old patches before posting this one.")	53	obsolete_patches = make_option("--no-obsolete", action="store_false", dest="obsolete_patches", default=True, help="Do not obsolete old patches before posting this one.")

Lines 1-4 a/Tools/Scripts/webkitpy/tool/steps/steps_unittest.py_sec1
- a/Tools/Scripts/webkitpy/tool/steps/steps_unittest.py -2 / +105 lines
1	# Copyright (C) 2010 Google Inc. All rights reserved.	1	# Copyright (C) 2010 Google Inc. All rights reserved.
		2	# Copyright (C) 2017 Apple Inc. All rights reserved.
2	#	3	#
3	# Redistribution and use in source and binary forms, with or without	4	# Redistribution and use in source and binary forms, with or without
4	# modification, are permitted provided that the following conditions are	5	# modification, are permitted provided that the following conditions are
Lines 28-33 a/Tools/Scripts/webkitpy/tool/steps/steps_unittest.py_sec2
28		29
29	import unittest	30	import unittest
30		31
		32	from webkitpy.common.system.executive import ScriptError
31	from webkitpy.common.system.outputcapture import OutputCapture	33	from webkitpy.common.system.outputcapture import OutputCapture
32	from webkitpy.common.config.ports import DeprecatedPort	34	from webkitpy.common.config.ports import DeprecatedPort
33	from webkitpy.tool.mocktool import MockOptions, MockTool	35	from webkitpy.tool.mocktool import MockOptions, MockTool
Lines 111-117 MOCK run_and_throw_if_fail: ['Tools/Scripts/test-webkitpy'], cwd=/mock-checkout a/Tools/Scripts/webkitpy/tool/steps/steps_unittest.py_sec3
111	Running Perl unit tests	113	Running Perl unit tests
112	MOCK run_and_throw_if_fail: ['Tools/Scripts/test-webkitperl'], cwd=/mock-checkout	114	MOCK run_and_throw_if_fail: ['Tools/Scripts/test-webkitperl'], cwd=/mock-checkout
113	Running JavaScriptCore tests	115	Running JavaScriptCore tests
114	MOCK run_and_throw_if_fail: ['Tools/Scripts/run-javascriptcore-tests'], cwd=/mock-checkout	116	MOCK run_and_throw_if_fail: ['Tools/Scripts/run-javascriptcore-tests', '--no-fail-fast', '--json-output=jsc_test_results.json'], cwd=/mock-checkout
115	Running bindings generation tests	117	Running bindings generation tests
116	MOCK run_and_throw_if_fail: ['Tools/Scripts/run-bindings-tests'], cwd=/mock-checkout	118	MOCK run_and_throw_if_fail: ['Tools/Scripts/run-bindings-tests'], cwd=/mock-checkout
117	Running run-webkit-tests	119	Running run-webkit-tests
Lines 133-142 MOCK run_and_throw_if_fail: ['Tools/Scripts/test-webkitpy'], cwd=/mock-checkout a/Tools/Scripts/webkitpy/tool/steps/steps_unittest.py_sec4
133	Running Perl unit tests	135	Running Perl unit tests
134	MOCK run_and_throw_if_fail: ['Tools/Scripts/test-webkitperl'], cwd=/mock-checkout	136	MOCK run_and_throw_if_fail: ['Tools/Scripts/test-webkitperl'], cwd=/mock-checkout
135	Running JavaScriptCore tests	137	Running JavaScriptCore tests
136	MOCK run_and_throw_if_fail: ['Tools/Scripts/run-javascriptcore-tests'], cwd=/mock-checkout	138	MOCK run_and_throw_if_fail: ['Tools/Scripts/run-javascriptcore-tests', '--no-fail-fast', '--json-output=jsc_test_results.json'], cwd=/mock-checkout
137	Running bindings generation tests	139	Running bindings generation tests
138	MOCK run_and_throw_if_fail: ['Tools/Scripts/run-bindings-tests'], cwd=/mock-checkout	140	MOCK run_and_throw_if_fail: ['Tools/Scripts/run-bindings-tests'], cwd=/mock-checkout
139	Running run-webkit-tests	141	Running run-webkit-tests
140	MOCK run_and_throw_if_fail: ['Tools/Scripts/run-webkit-tests', '--debug', '--quiet'], cwd=/mock-checkout	142	MOCK run_and_throw_if_fail: ['Tools/Scripts/run-webkit-tests', '--debug', '--quiet'], cwd=/mock-checkout
141	"""	143	"""
142	OutputCapture().assert_outputs(self, step.run, [{}], expected_logs=expected_logs)	144	OutputCapture().assert_outputs(self, step.run, [{}], expected_logs=expected_logs)
		145
		146	def test_runtests_jsc(self):
		147	mock_options = self._step_options()
		148	mock_options.non_interactive = False
		149	mock_options.build_style = "release"
		150	mock_options.group = "jsc"
		151	step = steps.RunTests(MockTool(log_executive=True), mock_options)
		152	tool = MockTool(log_executive=True)
		153	# FIXME: We shouldn't use a real port-object here, but there is too much to mock at the moment.
		154	tool._deprecated_port = DeprecatedPort()
		155	step = steps.RunTests(tool, mock_options)
		156	expected_logs = """MOCK run_and_throw_if_fail: ['Tools/Scripts/run-javascriptcore-tests', '--no-fail-fast', '--json-output=jsc_test_results.json'], cwd=/mock-checkout
		157	"""
		158	OutputCapture().assert_outputs(self, step.run, [{}], expected_logs=expected_logs)
		159
		160	def test_runtests_jsc_debug(self):
		161	mock_options = self._step_options()
		162	mock_options.non_interactive = False
		163	mock_options.build_style = "debug"
		164	mock_options.group = "jsc"
		165	tool = MockTool(log_executive=True)
		166	# FIXME: We shouldn't use a real port-object here, but there is too much to mock at the moment.
		167	tool._deprecated_port = DeprecatedPort()
		168	step = steps.RunTests(tool, mock_options)
		169	expected_logs = """MOCK run_and_throw_if_fail: ['Tools/Scripts/run-javascriptcore-tests', '--no-fail-fast', '--json-output=jsc_test_results.json', '--debug'], cwd=/mock-checkout
		170	"""
		171	OutputCapture().assert_outputs(self, step.run, [{}], expected_logs=expected_logs)
		172
		173	def test_build_jsc_debug(self):
		174	mock_options = self._step_options()
		175	mock_options.non_interactive = False
		176	mock_options.build_style = "debug"
		177	mock_options.build = True
		178	mock_options.architecture = True
		179	mock_options.group = "jsc"
		180	tool = MockTool(log_executive=True)
		181	# FIXME: We shouldn't use a real port-object here, but there is too much to mock at the moment.
		182	tool._deprecated_port = DeprecatedPort()
		183	step = steps.Build(tool, mock_options)
		184	expected_logs = """Building WebKit
		185	MOCK run_and_throw_if_fail: ['Tools/Scripts/build-jsc', '--debug', 'ARCHS=True'], cwd=/mock-checkout, env={'LC_ALL': 'C', 'TERM': 'none', 'MOCK_ENVIRON_COPY': '1'}
		186	"""
		187	OutputCapture().assert_outputs(self, step.run, [{}], expected_logs=expected_logs)
		188
		189	def test_build_jsc(self):
		190	mock_options = self._step_options()
		191	mock_options.non_interactive = False
		192	mock_options.build_style = "release"
		193	mock_options.build = True
		194	mock_options.architecture = True
		195	mock_options.group = "jsc"
		196	tool = MockTool(log_executive=True)
		197	# FIXME: We shouldn't use a real port-object here, but there is too much to mock at the moment.
		198	tool._deprecated_port = DeprecatedPort()
		199	step = steps.Build(tool, mock_options)
		200	expected_logs = """Building WebKit
		201	MOCK run_and_throw_if_fail: ['Tools/Scripts/build-jsc', '--release', 'ARCHS=True'], cwd=/mock-checkout, env={'LC_ALL': 'C', 'TERM': 'none', 'MOCK_ENVIRON_COPY': '1'}
		202	"""
		203	OutputCapture().assert_outputs(self, step.run, [{}], expected_logs=expected_logs)
		204
		205	def test_patch_relevant(self):
		206	self.maxDiff = None
		207	mock_options = self._step_options()
		208	tool = MockTool(log_executive=True)
209	tool.scm()._mockChangedFiles = ["JSTests/MockFile1", "ChangeLog"]
210	# FIXME: We shouldn't use a real port-object here, but there is too much to mock at the moment.
211	tool._deprecated_port = DeprecatedPort()
212	step = steps.CheckPatchRelevance(tool, mock_options)
213	expected_logs = """Checking relevance of patch
214	This patch has relevant changes.
215	"""
216	OutputCapture().assert_outputs(self, step.run, [{}], expected_logs=expected_logs)
217
218	def test_patch_relevant_jsc(self):
219	self.maxDiff = None
220	mock_options = self._step_options()
221	mock_options.group = "jsc"
222	tool = MockTool(log_executive=True)
223	tool.scm()._mockChangedFiles = ["JSTests/MockFile1", "ChangeLog"]
224	# FIXME: We shouldn't use a real port-object here, but there is too much to mock at the moment.
225	tool._deprecated_port = DeprecatedPort()
226	step = steps.CheckPatchRelevance(tool, mock_options)
227	expected_logs = """Checking relevance of patch
228	This patch has relevant changes.
229	"""
230	OutputCapture().assert_outputs(self, step.run, [{}], expected_logs=expected_logs)
231
232	def test_patch_not_relevant_jsc(self):
233	self.maxDiff = None
234	mock_options = self._step_options()
235	mock_options.group = "jsc"
236	tool = MockTool(log_executive=True)
237	tool.scm()._mockChangedFiles = ["Tools/ChangeLog", "Tools/Scripts/webkitpy/tool/steps/steps_unittest.py"]
238	# FIXME: We shouldn't use a real port-object here, but there is too much to mock at the moment.
239	tool._deprecated_port = DeprecatedPort()
240	step = steps.CheckPatchRelevance(tool, mock_options)
241	expected_logs = """Checking relevance of patch
242	This patch does not have relevant changes.
243	"""
244	with self.assertRaises(ScriptError):
245	OutputCapture().assert_outputs(self, step.run, [{}], expected_logs=expected_logs)