Attachment #301069 for bug #162458

View Details Raw Unified Return to Bug 162458

| Differences between



2017-02-08  Srinivasan Vijayaraghavan  <svijayaraghavan@apple.com>

        EWS should run JavaScriptCore tests
        https://bugs.webkit.org/show_bug.cgi?id=162458

        Reviewed by NOBODY (OOPS!).

        * QueueStatusServer/config/queues.py: Add jsc-ews queue
        * QueueStatusServer/handlers/statusbubble.py:
        (StatusBubble._should_show_bubble_for): Add logic to hide jsc-ews bubble if the patch does not touch jsc.
        * Scripts/webkitpy/common/checkout/scm/scm_mock.py:
        (MockSCM.__init__): Add _mockChangedFiles variable.
        (MockSCM.changed_files): Change to use _mockChangedFiles variables.
        * Scripts/webkitpy/common/config/ews.json: Add config info for JSC EWS.
        * Scripts/webkitpy/common/config/ports.py: Add support for JSC EWS in Mac Port.
        (DeprecatedPort.build_jsc_command): Added - command to build JSC only (quicker than building all of WebKit).
        (DeprecatedPort.run_javascriptcore_tests_command): Allow JSC EWS to only run JSC tests.
        (MacPort.run_webkit_tests_command): Check for JSC.
        * Scripts/webkitpy/common/config/ports_mock.py:
        (MockPort.run_javascriptcore_tests_command): Add build_style argument.
        * Scripts/webkitpy/common/config/ports_unittest.py:
        (DeprecatedPortTest.test_mac_port): Add unit tests for build-jsc shell command.
        * Scripts/webkitpy/common/net/jsctestresults.py: Added.
        (JSCTestResults): Added.
        (JSCTestResults.intersection): Return a JSCTestResults object with failures common to both input objects.
        (JSCTestResults.results_from_string): Use json library to parse results.
        (JSCTestResults.__init__): Initialize members.
        (JSCTestResults.equals): This enables unit testing.
        (JSCTestResults.is_subset): Checks if one set of failures is a subset of another.
        (JSCTestResults.stress_test_failures): Getter.
        (JSCTestResults.all_api_tests_passed): Getter.
        (JSCTestResults.all_passed): Getter.
        (JSCTestResults.failing_test_results): Getter. Some of these exist for interopability reasons.
        (JSCTestResults.test_results): Getter.
        (JSCTestResults.failing_tests): Getter.
        (JSCTestResults.did_exceed_test_failure_limit): Getter.
        * Scripts/webkitpy/common/net/jsctestresults_unittest.py: Added.
        (JSCTestResultsTest): Class to test JSCTestResults.
        (JSCTestResultsTest.test_results_from_string): Tests parsing.
        (JSCTestResultsTest.test_intersection_api_tests): Unit test for intersection() class method.
        (JSCTestResultsTest.test_intersection_stress_tests): Unit test for intersection() class method.
        (JSCTestResultsTest.test_intersection_general_case): Unit test for intersection() class method.
        * Scripts/webkitpy/tool/bot/earlywarningsystemtask.py:
        (EarlyWarningSystemTask.run): Add check_patch_relevance step.
        * Scripts/webkitpy/tool/bot/jscews_unittest.py: Added.
        (MockPatchAnalysisTask): Mocked-out version of PatchAnalysisTask that doesn't run commands.
        (MockPatchAnalysisTask.__init__): Sets attributes.
        (MockPatchAnalysisTask._test): Override to not run command.
        (MockPatchAnalysisTask._build_and_test_without_patch): Override to not run command.
        (MockPatchAnalysisTask.validate): Assume mocked patch is valid for purposes of testing retry logic.
        (MockPatchAnalysisTask.test_run_count): Specific to the mocked version, to test retry logic.
        (MockJSCEarlyWarningSystem): Mocked-out version of AbstarctEarlyWarningSystem so we can provide test results.
        (MockJSCEarlyWarningSystem.__init__): Sets attributes, also sets group to jsc.
        (MockJSCEarlyWarningSystem.test_results): Returns test results provided by us, instead of using a JSON reader.
        (JSCEarlyWarningSystemTest): Class to test retry logic in below situations.
        (JSCEarlyWarningSystemTest._create_task): Helper function to abstract out common code.
        (JSCEarlyWarningSystemTest._results_indicate_all_passed): False if input is None or has failures, else True.
        (JSCEarlyWarningSystemTest.test_success_case): Clean patch on clean tree.
        (JSCEarlyWarningSystemTest.test_test_failure): Failed patch on clean tree.
        (JSCEarlyWarningSystemTest.test_fix): Patch that fixes all tree redness.
        (JSCEarlyWarningSystemTest.test_ineffective_patch): Patch that has same failures as tree.
        (JSCEarlyWarningSystemTest.test_partially_effective_patch): Patch fixes some failures but adds no new failures.
        (JSCEarlyWarningSystemTest.test_different_test_failures_in_patch_and_tree): Patch has some failures not in tree.
        (JSCEarlyWarningSystemTest.test_first_results_could_not_be_read): Patch results not readable.
        (JSCEarlyWarningSystemTest.test_second_results_could_not_be_read): Patch results not readable on second run.
        (JSCEarlyWarningSystemTest.test_clean_results_could_not_be_read): Results from clean tree not readable.
        (JSCEarlyWarningSystemTest.test_flaky_results_on_clean_tree_pass): Patch has one flake and no failures.
        (JSCEarlyWarningSystemTest.test_flaky_results_on_clean_tree_pass_v2): Patch has one flake and no failures.
        (JSCEarlyWarningSystemTest.test_flaky_results_on_clean_tree_failure): Patch has flakes and failed tests.
        (JSCEarlyWarningSystemTest.test_flaky_results_on_red_tree_pass): Patch has same failures as tree, plus a flake.
        * Scripts/webkitpy/tool/bot/jsctestresultsreader.py: Added.
        (JSCTestResultsReader): Reads results file.
        (JSCTestResultsReader.__init__): Sets attributes.
        (JSCTestResultsReader._read_file_contents): Reads file.
        (JSCTestResultsReader._create_jsc_test_results): Creates a JSCTestResults object.
        (JSCTestResultsReader.results): Invokes _create_jsc_test_results to return a JSCTestResults object.
        * Scripts/webkitpy/tool/bot/patchanalysistask.py:
        (PatchIsNotApplicable): Exception for when patch doesn't have relevant changes.
        (PatchAnalysisTask._check_patch_relevance): Added.
        (PatchAnalysisTask._build): Check for JSC.
        (PatchAnalysisTask._build_without_patch): Check for JSC.
        (PatchAnalysisTask._test): Check for JSC.
        (PatchAnalysisTask._build_and_test_without_patch): Check for JSC.
        (PatchAnalysisTask._test_patch): Add retry logic for JSC.
        * Scripts/webkitpy/tool/commands/download.py:
        (CheckPatchRelevance): Add check-patch-relevance command.
        * Scripts/webkitpy/tool/commands/earlywarningsystem.py:
        (AbstractEarlyWarningSystem._create_task): Abstract out to enable mocking.
        (AbstractEarlyWarningSystem.begin_work_queue): Use JSCTestResultsReader not LayoutTestResultsReader in JSC EWS.
        (AbstractEarlyWarningSystem.review_patch): Handle PatchIsApplicable
        (AbstractEarlyWarningSystem.test_results): _layout_test_results_reader -> _test_results_reader.
        (AbstractEarlyWarningSystem.archive_last_test_results): _layout_test_results_reader -> _test_results_reader.
        (AbstractEarlyWarningSystem.group): This attribute determines the type of EWS (eg. JSC).
        (AbstractEarlyWarningSystem.load_ews_classes): Add _group, and make classes of type cls to enable mocking.
        * Scripts/webkitpy/tool/commands/earlywarningsystem_unittest.py:
        (AbstractEarlyWarningSystemTest.test_failing_tests_message.TestEWS): Add _group.
        (AbstractEarlyWarningSystemTest.test_failing_jsc_tests_message.TestEWS): Added test for jsc failures message.
        (AbstractEarlyWarningSystemTest): Add _group variable.
        (EarlyWarningSystemTest._default_expected_logs): Add check-patch-relevance step, inconclusive logs, and group.
        (EarlyWarningSystemTest._test_ews): Add logs_are_conclusive option to pass through to default_expected_logs().
        (EarlyWarningSystemTest.test_inconclusive_test_results): Test not removing patch from queue if not conclusive.
        (MockAbstractEarlyWarningSystem): Added.
        (MockAbstractEarlyWarningSystem.test_results): Simulates inconclusive results.
        (MockAbstractEarlyWarningSystem._create_task): Use MockEarlyWarningSystemTask (not EarlyWarningSystemTask).
        (MockEarlyWarningSystemTask): Added.
        (MockEarlyWarningSystemTask._test_patch): Simulates running tests but not getting a conclusive result.
        * Scripts/webkitpy/tool/steps/__init__.py: Add CheckPatchRelevance import.
        * Scripts/webkitpy/tool/steps/build.py:
        (Build.options): Check for JSC.
        (Build.build): Check for JSC.
        (Build.run): Check for JSC.
        * Scripts/webkitpy/tool/steps/checkpatchrelevance.py: Added.
        (CheckPatchRelevance): Added.
        (CheckPatchRelevance._does_contain_change_in_paths): Abstract function to perform regex matching.
        (CheckPatchRelevance.run): Check if changed files in patch belong in certain folders.
        * Scripts/webkitpy/tool/steps/options.py:
        (Options): Add --group command line option.
        * Scripts/webkitpy/tool/steps/runtests.py:
        (RunTests.options): Add group.
        (RunTests.run): Check for JSC.
        (RunTests._run_webkit_tests): Check for JSC.
        (RunTests): Add _group attribute.
        (RunTests._run_javascriptcore_tests): New.
        * Scripts/webkitpy/tool/steps/steps_unittest.py: Unit tests.

2017-02-03  Simon Fraser  <simon.fraser@apple.com>

        Correctly set the "inWindow" flag for TileControllers that aren't the page tiles, and clarify "usingTiledBacking" logic

Lines 1-4 a/Tools/QueueStatusServer/config/queues.py_sec1
- a/Tools/QueueStatusServer/config/queues.py -1 / +2 lines
1	# Copyright (C) 2014 Apple Inc. All rights reserved.	1	# Copyright (C) 2014, 2017 Apple Inc. All rights reserved.
2	# Copyright (C) 2013 Google Inc. All rights reserved.	2	# Copyright (C) 2013 Google Inc. All rights reserved.
3	#	3	#
4	# Redistribution and use in source and binary forms, with or without	4	# Redistribution and use in source and binary forms, with or without
Lines 36-41 all_queue_names = [ a/Tools/QueueStatusServer/config/queues.py_sec2
36	"gtk-wk2-ews",	36	"gtk-wk2-ews",
37	"ios-ews",	37	"ios-ews",
38	"ios-sim-ews",	38	"ios-sim-ews",
		39	"jsc-ews",
39	"mac-ews",	40	"mac-ews",
40	"mac-wk2-ews",	41	"mac-wk2-ews",
41	"mac-debug-ews",	42	"mac-debug-ews",



# Copyright (C) 2009 Google Inc. All rights reserved.
# Copyright (C) 2017 Apple Inc. All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are

        return bubble

    def _should_show_bubble_for(self, attachment, queue):
        # Any pending queue is shown.
        if attachment.position_in_queue(queue):
            return True

        # EWS queues are also shown when complete
        # But JSC EWS bubble is not shown if the patch is not relevant to JSC.
        if not queue.is_ews():
            return False

        status = attachment.status_for_queue(queue)
        if queue.name() == "jsc-ews":
            return not status.message.startswith("Patch not applicable")
        return bool(status)

    def _build_bubbles_for_attachment(self, attachment):
        show_submit_to_ews = True



# Copyright (C) 2011 Google Inc. All rights reserved.
# Copyright (C) 2017 Apple Inc. All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are

        self.added_paths = set()
        self._filesystem = filesystem or MockFileSystem()
        self._executive = executive or MockExecutive()
        self._mockChangedFiles = ["MockFile1"]

    def add(self, destination_path):
        self.add_list([destination_path])

        return self._filesystem.join(self.checkout_root, *comps)

    def changed_files(self, git_commit=None):
        return self._mockChangedFiles

    def changed_files_for_revision(self, revision):
        return ["MockFile1"]

Lines 50-54 a/Tools/Scripts/webkitpy/common/config/ews.json_sec1
- a/Tools/Scripts/webkitpy/common/config/ews.json +6 lines
50	"port": "mac",	50	"port": "mac",
51	"name": "mac-32bit-ews",	51	"name": "mac-32bit-ews",
52	"architecture": "i386"	52	"architecture": "i386"
		53	},
		54	"JSC EWS": {
		55	"port": "mac",
		56	"name": "jsc-ews",
		57	"group": "jsc",
		58	"runTests": true
53	}	59	}
54	}	60	}



# Copyright (C) 2009, Google Inc. All rights reserved.
# Copyright (C) 2013 Nokia Corporation and/or its subsidiary(-ies).
# Copyright (C) 2015, 2017 Apple Inc. All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are

            command.append("--release")
        return command

    def build_jsc_command(self, build_style=None):
        command = self.script_shell_command("build-jsc")
        if build_style == "debug":
            command.append("--debug")
        if build_style == "release":
            command.append("--release")
        return command

    def run_javascriptcore_tests_command(self, build_style=None):
        command = self.script_shell_command("run-javascriptcore-tests")
        command.append("--no-fail-fast")
        command.append("--json-output=jsc_test_results.json")
        if (build_style == "debug"):
            command.append("--debug")
        return command

    def run_webkit_tests_command(self, build_style=None):
        command = self.script_shell_command("run-webkit-tests")

class MacPort(DeprecatedPort):
    port_flag_name = "mac"

    def run_webkit_tests_command(self, build_style=None, test_suite=None):
        if (test_suite == "jsc"):
            return super(MacPort, self).run_javascriptcore_tests_command(build_style)

        command = super(MacPort, self).run_webkit_tests_command(build_style)
        command.append("--dump-render-tree")
        return command



# Copyright (C) 2011 Google Inc. All rights reserved.
# Copyright (C) 2017 Apple Inc. All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are

    def run_perl_unittests_command(self):
        return ['mock-test-webkitperl']

    def run_javascriptcore_tests_command(self, build_style=None):
        return ['mock-run-javacriptcore-tests']

    def run_webkit_tests_command(self, build_style=None):

Lines 1-4 a/Tools/Scripts/webkitpy/common/config/ports_unittest.py_sec1
- a/Tools/Scripts/webkitpy/common/config/ports_unittest.py +5 lines
1	# Copyright (c) 2009, Google Inc. All rights reserved.	1	# Copyright (c) 2009, Google Inc. All rights reserved.
		2	# Copyright (C) 2017 Apple Inc. All rights reserved.
2	#	3	#
3	# Redistribution and use in source and binary forms, with or without	4	# Redistribution and use in source and binary forms, with or without
4	# modification, are permitted provided that the following conditions are	5	# modification, are permitted provided that the following conditions are
Lines 39-44 class DeprecatedPortTest(unittest.TestCase): a/Tools/Scripts/webkitpy/common/config/ports_unittest.py_sec2
39	self.assertEqual(MacPort().build_webkit_command(build_style="debug"), DeprecatedPort().script_shell_command("build-webkit") + ["--debug"])	40	self.assertEqual(MacPort().build_webkit_command(build_style="debug"), DeprecatedPort().script_shell_command("build-webkit") + ["--debug"])
40	self.assertEqual(MacPort().build_webkit_command(build_style="release"), DeprecatedPort().script_shell_command("build-webkit") + ["--release"])	41	self.assertEqual(MacPort().build_webkit_command(build_style="release"), DeprecatedPort().script_shell_command("build-webkit") + ["--release"])
41		42
		43	self.assertEqual(MacPort().build_jsc_command(), DeprecatedPort().script_shell_command("build-jsc"))
		44	self.assertEqual(MacPort().build_jsc_command(build_style="release"), DeprecatedPort().script_shell_command("build-jsc") + ["--release"])
		45	self.assertEqual(MacPort().build_jsc_command(build_style="debug"), DeprecatedPort().script_shell_command("build-jsc") + ["--debug"])
		46
42	def test_gtk_wk2_port(self):	47	def test_gtk_wk2_port(self):
43	self.assertEqual(GtkWK2Port().flag(), "--port=gtk-wk2")	48	self.assertEqual(GtkWK2Port().flag(), "--port=gtk-wk2")
44	self.assertEqual(GtkWK2Port().run_webkit_tests_command(), DeprecatedPort().script_shell_command("run-webkit-tests") + ["--gtk"])	49	self.assertEqual(GtkWK2Port().run_webkit_tests_command(), DeprecatedPort().script_shell_command("run-webkit-tests") + ["--gtk"])



# Copyright (C) 2017 Apple Inc. All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions
# are met:
# 1.  Redistributions of source code must retain the above copyright
#     notice, this list of conditions and the following disclaimer.
# 2.  Redistributions in binary form must reproduce the above copyright
#     notice, this list of conditions and the following disclaimer in the
#     documentation and/or other materials provided with the distribution.
#
# THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS'' AND
# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
# DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS BE LIABLE FOR
# ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
# SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

import json
import logging

_log = logging.getLogger(__name__)


class JSCTestResults(object):
    @classmethod
    def intersection(cls, first, second):
        intersection_api_tests_passed = first._all_api_tests_passed or second._all_api_tests_passed
        intersection_stress_test_failures = list(set(first._stress_test_failures) & set(second._stress_test_failures))
        return cls(intersection_api_tests_passed, intersection_stress_test_failures)

    @classmethod
    def results_from_string(cls, string):
        if not string:
            return None

        try:
            parsed_results = json.loads(string)
        except ValueError:
            return None

        if 'allApiTestsPassed' not in parsed_results or 'stressTestFailures' not in parsed_results:
            return None

        return cls(parsed_results['allApiTestsPassed'], parsed_results['stressTestFailures'])

    def __init__(self, all_api_tests_passed, stress_test_failures):
        self._all_api_tests_passed = all_api_tests_passed
        self._stress_test_failures = stress_test_failures

        self._test_results = stress_test_failures[:]
        if not self._all_api_tests_passed:
            self._test_results.append("apiTests")

    def equals(self, other):
        return self._all_api_tests_passed == other._all_api_tests_passed and \
               set(self._stress_test_failures) == set(other._stress_test_failures)

    def is_subset(self, other):
        return set(self.test_results()) <= set(other.test_results())

    def stress_test_failures(self):
        return self._stress_test_failures

    def all_api_tests_passed(self):
        return self._all_api_tests_passed

    def all_passed(self):
        return self._all_api_tests_passed and not self._stress_test_failures

    def failing_test_results(self):
        return self.stress_test_failures()

    def test_results(self):
        return self._test_results

    def failing_tests(self):
        return self._test_results

    # No defined failure limit for JSC.
    def did_exceed_test_failure_limit(self):
        return False



# Copyright (C) 2017 Apple Inc. All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions
# are met:
# 1.  Redistributions of source code must retain the above copyright
#     notice, this list of conditions and the following disclaimer.
# 2.  Redistributions in binary form must reproduce the above copyright
#     notice, this list of conditions and the following disclaimer in the
#     documentation and/or other materials provided with the distribution.
#
# THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS'' AND
# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
# DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS BE LIABLE FOR
# ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
# SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

import unittest

from webkitpy.common.net.jsctestresults import JSCTestResults


class JSCTestResultsTest(unittest.TestCase):
    def test_results_from_string(self):
        none_item = None
        empty_string = ""
        invalid_json = '{"allApiTestsPassed":'
        incomplete_json_v1 = '{"allApiTestsPassed": true}'
        incomplete_json_v2 = '{"stressTestFailures":[]}'
        self.assertEqual(None, JSCTestResults.results_from_string(none_item))
        self.assertEqual(None, JSCTestResults.results_from_string(empty_string))
        self.assertEqual(None, JSCTestResults.results_from_string(invalid_json))
        self.assertEqual(None, JSCTestResults.results_from_string(incomplete_json_v1))
        self.assertEqual(None, JSCTestResults.results_from_string(incomplete_json_v2))

        no_failures_string = '{"allApiTestsPassed": true, "stressTestFailures":[]}'
        no_failures_results = JSCTestResults(True, [])
        self.assertTrue(no_failures_results.equals(JSCTestResults.results_from_string(no_failures_string)))

        api_test_failures_string = '{"allApiTestsPassed": false, "stressTestFailures":[]}'
        api_test_failures_results = JSCTestResults(False, [])
        self.assertTrue(api_test_failures_results.equals(JSCTestResults.results_from_string(api_test_failures_string)))

        many_failures_string = '{"allApiTestsPassed": false, "stressTestFailures":["es6.yaml/es6/typed_arrays_Int16Array.js.default", "es6.yaml/es6/typed_arrays_Int8Array.js.default"]}'
        many_failures_results = JSCTestResults(False, ["es6.yaml/es6/typed_arrays_Int16Array.js.default", "es6.yaml/es6/typed_arrays_Int8Array.js.default"])
        self.assertTrue(many_failures_results.equals(JSCTestResults.results_from_string(many_failures_string)))

        self.assertFalse(no_failures_results == api_test_failures_results)
        self.assertFalse(api_test_failures_results == many_failures_results)

    def test_intersection_api_tests(self):
        results1 = JSCTestResults(False, [])
        results2 = JSCTestResults(True, [])

        expected_intersection = JSCTestResults(True, [])
        self.assertTrue(expected_intersection.equals(JSCTestResults.intersection(results1, results2)))

    def test_intersection_stress_tests(self):
        results1 = JSCTestResults(True, ['failure1', 'failure2'])
        results2 = JSCTestResults(True, ['failure1', 'failure3'])

        expected_intersection = JSCTestResults(True, ['failure1'])
        self.assertTrue(expected_intersection.equals(JSCTestResults.intersection(results1, results2)))

    def test_intersection_general_case(self):
        results1 = JSCTestResults(True, ['failure1', 'failure2'])
        results2 = JSCTestResults(False, ['failure1'])

        expected_intersection = JSCTestResults(True, ['failure1'])
        self.assertTrue(expected_intersection.equals(JSCTestResults.intersection(results1, results2)))

Lines 1-4 a/Tools/Scripts/webkitpy/tool/bot/earlywarningsystemtask.py_sec1
- a/Tools/Scripts/webkitpy/tool/bot/earlywarningsystemtask.py -1 / +4 lines
1	# Copyright (c) 2011 Google Inc. All rights reserved.	1	# Copyright (c) 2011 Google Inc. All rights reserved.
		2	# Copyright (C) 2017 Apple Inc. All rights reserved.
2	#	3	#
3	# Redistribution and use in source and binary forms, with or without	4	# Redistribution and use in source and binary forms, with or without
4	# modification, are permitted provided that the following conditions are	5	# modification, are permitted provided that the following conditions are
Lines 26-32 a/Tools/Scripts/webkitpy/tool/bot/earlywarningsystemtask.py_sec2
26	# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE	27	# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
27	# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.	28	# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
28		29
29	from webkitpy.tool.bot.patchanalysistask import PatchAnalysisTask, PatchAnalysisTaskDelegate, UnableToApplyPatch, PatchIsNotValid	30	from webkitpy.tool.bot.patchanalysistask import PatchAnalysisTask, PatchAnalysisTaskDelegate, UnableToApplyPatch, PatchIsNotValid, PatchIsNotApplicable
30		31
31		32
32	class EarlyWarningSystemTaskDelegate(PatchAnalysisTaskDelegate):	33	class EarlyWarningSystemTaskDelegate(PatchAnalysisTaskDelegate):
Lines 58-63 class EarlyWarningSystemTask(PatchAnalysisTask): a/Tools/Scripts/webkitpy/tool/bot/earlywarningsystemtask.py_sec3
58	return False	59	return False
59	if not self._apply():	60	if not self._apply():
60	raise UnableToApplyPatch(self._patch)	61	raise UnableToApplyPatch(self._patch)
		62	if not self._check_patch_relevance():
		63	raise PatchIsNotApplicable(self._patch)
61	if not self._build():	64	if not self._build():
62	if not self._build_without_patch():	65	if not self._build_without_patch():
63	return False	66	return False



# Copyright (C) 2017 Apple Inc. All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions
# are met:
# 1.  Redistributions of source code must retain the above copyright
#     notice, this list of conditions and the following disclaimer.
# 2.  Redistributions in binary form must reproduce the above copyright
#     notice, this list of conditions and the following disclaimer in the
#     documentation and/or other materials provided with the distribution.
#
# THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS'' AND
# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
# DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS BE LIABLE FOR
# ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
# SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

import logging
import unittest

from webkitpy.common.net.jsctestresults import JSCTestResults
from webkitpy.common.system.executive import ScriptError
from webkitpy.tool.bot.patchanalysistask import *
from webkitpy.tool.commands.earlywarningsystem import AbstractEarlyWarningSystem
from webkitpy.tool.mocktool import MockTool

_log = logging.getLogger(__name__)


class MockPatchAnalysisTask(PatchAnalysisTask):
    def __init__(self, delegate, patch, patches_passed_all_tests):
        self._delegate = delegate
        self._patch = patch
        self._script_error = None
        self._results_archive_from_patch_test_run = None
        self._results_from_patch_test_run = None
        self.error = None
        self._patches_passed_all_tests = patches_passed_all_tests
        self._test_run_count = 0
        self.failure_status_id = 42

    def _test(self):
        self._test_run_count += 1

        if self._patches_passed_all_tests.pop() == True:
            return True
        else:
            self._script_error = ScriptError("Regression test")
            return False

    def _build_and_test_without_patch(self):
        self._test_run_count += 1
        return True

    def validate(self):
        return True

    def test_run_count(self):
        return self._test_run_count


# This is the delegate to be used with MockPatchAnalysisTask, above.
class MockJSCEarlyWarningSystem(AbstractEarlyWarningSystem):
    def __init__(self, first_test_results, second_test_results, clean_test_results):
        AbstractEarlyWarningSystem.__init__(self)
        self._group = "jsc"
        self._results_in_order = [clean_test_results, second_test_results, first_test_results]

    def test_results(self):
        return self._results_in_order.pop()


class JSCEarlyWarningSystemTest(unittest.TestCase):
    def _results_indicate_all_passed(self, results):
        if results == None:
            return False
        return results.all_passed()

    def _create_task(self, first_test_results, second_test_results, clean_test_results):
        queue = MockJSCEarlyWarningSystem(first_test_results, second_test_results, clean_test_results)
        tool = MockTool(log_executive=True)
        patch = tool.bugs.fetch_attachment(10000)
        patches_passed_all_tests = map(self._results_indicate_all_passed, [second_test_results, first_test_results])
        return MockPatchAnalysisTask(queue, patch, patches_passed_all_tests)

    def test_success_case(self):
        first_test_results = JSCTestResults(True, [])
        second_test_results = JSCTestResults(True, [])
        clean_test_results = JSCTestResults(True, [])
        task = self._create_task(first_test_results, second_test_results, clean_test_results)

        return_value = task._test_patch()
        self.assertEqual(task.test_run_count(), 1)
        self.assertEqual(return_value, True)

    def test_test_failure(self):
        first_test_results = JSCTestResults(True, ['Fail.js'])
        second_test_results = JSCTestResults(True, ['Fail.js'])
        clean_test_results = JSCTestResults(True, [])
        task = self._create_task(first_test_results, second_test_results, clean_test_results)

        with self.assertRaises(ScriptError):
            return_value = task._test_patch()
        self.assertEqual(task.test_run_count(), 3)

    def test_fix(self):
        first_test_results = JSCTestResults(True, [])
        second_test_results = JSCTestResults(True, [])
        clean_test_results = JSCTestResults(True, ['Fail.js'])
        task = self._create_task(first_test_results, second_test_results, clean_test_results)

        return_value = task._test_patch()
        self.assertEqual(task.test_run_count(), 1)
        self.assertEqual(return_value, True)

    def test_ineffective_patch(self):
        first_test_results = JSCTestResults(False, ['failure1.js', 'failure2.js'])
        second_test_results = JSCTestResults(False, ['failure1.js', 'failure2.js'])
        clean_test_results = JSCTestResults(False, ['failure1.js', 'failure2.js'])
        task = self._create_task(first_test_results, second_test_results, clean_test_results)

        return_value = task._test_patch()
        self.assertEqual(task.test_run_count(), 3)
        self.assertEqual(return_value, True)

    def test_partially_effective_patch(self):
        first_test_results = JSCTestResults(True, ['failure2.js'])
        second_test_results = JSCTestResults(True, ['failure2.js'])
        clean_test_results = JSCTestResults(False, ['failure1.js', 'failure2.js'])
        task = self._create_task(first_test_results, second_test_results, clean_test_results)

        return_value = task._test_patch()
        self.assertEqual(task.test_run_count(), 3)
        self.assertEqual(return_value, True)

    def test_different_test_failures_in_patch_and_tree(self):
        first_test_results = JSCTestResults(False, [])
        second_test_results = JSCTestResults(False, [])
        clean_test_results = JSCTestResults(True, ['failure1.js', 'failure2.js'])
        task = self._create_task(first_test_results, second_test_results, clean_test_results)

        with self.assertRaises(ScriptError):
            return_value = task._test_patch()
        self.assertEqual(task.test_run_count(), 3)

    def test_first_results_could_not_be_read(self):
        first_test_results = None
        second_test_results = JSCTestResults(True, [])
        clean_test_results = JSCTestResults(True, [])
        task = self._create_task(first_test_results, second_test_results, clean_test_results)

        return_value = task._test_patch()
        self.assertEqual(task.test_run_count(), 1)
        self.assertEqual(return_value, False)

    def test_second_results_could_not_be_read(self):
        first_test_results = JSCTestResults(False, ['failure1.js', 'failure2.js'])
        second_test_results = None
        clean_test_results = JSCTestResults(True, [])
        task = self._create_task(first_test_results, second_test_results, clean_test_results)

        return_value = task._test_patch()
        self.assertEqual(task.test_run_count(), 2)
        self.assertEqual(return_value, False)

    def test_clean_results_could_not_be_read(self):
        first_test_results = JSCTestResults(True, ['failure2.js'])
        second_test_results = JSCTestResults(True, ['failure2.js'])
        clean_test_results = None
        task = self._create_task(first_test_results, second_test_results, clean_test_results)

        return_value = task._test_patch()
        self.assertEqual(task.test_run_count(), 3)
        self.assertEqual(return_value, False)

    def test_flaky_results_on_clean_tree_pass(self):
        first_test_results = JSCTestResults(True, ['failure2.js'])
        second_test_results = JSCTestResults(True, [])
        clean_test_results = JSCTestResults(True, [])
        task = self._create_task(first_test_results, second_test_results, clean_test_results)

        return_value = task._test_patch()
        self.assertEqual(return_value, True)
        self.assertEqual(task.test_run_count(), 2)

    def test_flaky_results_on_clean_tree_pass_v2(self):
        first_test_results = JSCTestResults(True, [])
        second_test_results = JSCTestResults(True, ['failure2.js'])
        clean_test_results = JSCTestResults(True, [])
        task = self._create_task(first_test_results, second_test_results, clean_test_results)

        return_value = task._test_patch()
        self.assertEqual(return_value, True)
        self.assertEqual(task.test_run_count(), 1)

    def test_flaky_results_on_clean_tree_failure(self):
        first_test_results = JSCTestResults(False, ['failure1.js', 'failure2.js'])
        second_test_results = JSCTestResults(True, ['failure2.js'])
        clean_test_results = JSCTestResults(True, [])
        task = self._create_task(first_test_results, second_test_results, clean_test_results)

        with self.assertRaises(ScriptError):
            return_value = task._test_patch()
        self.assertEqual(task.test_run_count(), 3)

    def test_flaky_results_on_red_tree_pass(self):
        first_test_results = JSCTestResults(True, ['failure1.js'])
        second_test_results = JSCTestResults(True, ['failure1.js', 'failure2.js'])
        clean_test_results = JSCTestResults(True, ['failure1.js'])
        task = self._create_task(first_test_results, second_test_results, clean_test_results)

        return_value = task._test_patch()
        self.assertEqual(return_value, True)
        self.assertEqual(task.test_run_count(), 3)



# Copyright (C) 2017 Apple Inc. All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions
# are met:
# 1.  Redistributions of source code must retain the above copyright
#     notice, this list of conditions and the following disclaimer.
# 2.  Redistributions in binary form must reproduce the above copyright
#     notice, this list of conditions and the following disclaimer in the
#     documentation and/or other materials provided with the distribution.
#
# THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS'' AND
# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
# DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS BE LIABLE FOR
# ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
# SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

import logging

from webkitpy.common.net.jsctestresults import JSCTestResults

_log = logging.getLogger(__name__)


class JSCTestResultsReader(object):
    def __init__(self, host, results_directory):
        self._host = host
        self._results_directory = results_directory

    def _read_file_contents(self, path):
        try:
            return self._host.filesystem.read_text_file(path)
        except (IOError, KeyError):  # File does not exist or can't be read.
            return None

    def _create_jsc_test_results(self):
        results_path = self._host.filesystem.join(self._results_directory, "jsc_test_results.json")
        contents = self._read_file_contents(results_path)
        return JSCTestResults.results_from_string(contents)

    def results(self):
        return self._create_jsc_test_results()

Lines 1-4 a/Tools/Scripts/webkitpy/tool/bot/patchanalysistask.py_sec1
- a/Tools/Scripts/webkitpy/tool/bot/patchanalysistask.py -62 / +135 lines
1	# Copyright (c) 2010 Google Inc. All rights reserved.	1	# Copyright (c) 2010 Google Inc. All rights reserved.
		2	# Copyright (C) 2017 Apple Inc. All rights reserved.
2	#	3	#
3	# Redistribution and use in source and binary forms, with or without	4	# Redistribution and use in source and binary forms, with or without
4	# modification, are permitted provided that the following conditions are	5	# modification, are permitted provided that the following conditions are
Lines 28-33 a/Tools/Scripts/webkitpy/tool/bot/patchanalysistask.py_sec2
28		29
29	from webkitpy.common.system.executive import ScriptError	30	from webkitpy.common.system.executive import ScriptError
30	from webkitpy.common.net.layouttestresults import LayoutTestResults	31	from webkitpy.common.net.layouttestresults import LayoutTestResults
		32	from webkitpy.common.net.jsctestresults import JSCTestResults
31		33
32		34
33	class UnableToApplyPatch(Exception):	35	class UnableToApplyPatch(Exception):
Lines 43-48 class PatchIsNotValid(Exception): a/Tools/Scripts/webkitpy/tool/bot/patchanalysistask.py_sec3
43	self.failure_message = failure_message	45	self.failure_message = failure_message
44		46
45		47
		48	class PatchIsNotApplicable(Exception):
		49	def __init__(self, patch):
		50	Exception.__init__(self)
		51	self.patch = patch
		52
46	class PatchAnalysisTaskDelegate(object):	53	class PatchAnalysisTaskDelegate(object):
47	def parent_command(self):	54	def parent_command(self):
48	raise NotImplementedError("subclasses must implement")	55	raise NotImplementedError("subclasses must implement")
Lines 122-149 class PatchAnalysisTask(object): a/Tools/Scripts/webkitpy/tool/bot/patchanalysistask.py_sec4
122	"Applied patch",	129	"Applied patch",
123	"Patch does not apply")	130	"Patch does not apply")
124		131
		132	def _check_patch_relevance(self):
		133	args = [
		134	"check-patch-relevance",
		135	]
		136
		137	if hasattr(self._delegate, 'group'):
		138	args.append("--group=%s" % self._delegate.group())
		139
		140	return self._run_command(
		141	args,
		142	"Checked relevance of patch",
		143	"Patch was not relevant")
		144
125	def _build(self):	145	def _build(self):
126	return self._run_command([	146	args = [
127	"build",	147	"build",
128	"--no-clean",	148	"--no-clean",
129	"--no-update",	149	"--no-update",
130	"--build-style=%s" % self._delegate.build_style(),	150	"--build-style=%s" % self._delegate.build_style(),
131	],	151	]
132	"Built patch",	152
133	"Patch does not build")	153	if hasattr(self._delegate, 'group'):
		154	args.append("--group=%s" % self._delegate.group())
		155
		156	return self._run_command(
		157	args,
		158	"Built patch",
		159	"Patch does not build")
134		160
135	def _build_without_patch(self):	161	def _build_without_patch(self):
136	return self._run_command([	162	args = [
137	"build",	163	"build",
138	"--force-clean",	164	"--force-clean",
139	"--no-update",	165	"--no-update",
140	"--build-style=%s" % self._delegate.build_style(),	166	"--build-style=%s" % self._delegate.build_style(),
141	],	167	]
142	"Able to build without patch",	168
143	"Unable to build without patch")	169	if hasattr(self._delegate, 'group'):
		170	args.append("--group=%s" % self._delegate.group())
		171
		172	return self._run_command(
		173	args,
		174	"Able to build without patch",
		175	"Unable to build without patch")
144		176
145	def _test(self):	177	def _test(self):
146	return self._run_command([	178	args = [
147	"build-and-test",	179	"build-and-test",
148	"--no-clean",	180	"--no-clean",
149	"--no-update",	181	"--no-update",
Lines 151-162 class PatchAnalysisTask(object): a/Tools/Scripts/webkitpy/tool/bot/patchanalysistask.py_sec5
151	"--test",	183	"--test",
152	"--non-interactive",	184	"--non-interactive",
153	"--build-style=%s" % self._delegate.build_style(),	185	"--build-style=%s" % self._delegate.build_style(),
154	],	186	]
155	"Passed tests",	187
156	"Patch does not pass tests")	188	if hasattr(self._delegate, 'group'):
		189	args.append("--group=%s" % self._delegate.group())
		190
		191	return self._run_command(
		192	args,
		193	"Passed tests",
		194	"Patch does not pass tests")
157		195
158	def _build_and_test_without_patch(self):	196	def _build_and_test_without_patch(self):
159	return self._run_command([	197	args = [
160	"build-and-test",	198	"build-and-test",
161	"--force-clean",	199	"--force-clean",
162	"--no-update",	200	"--no-update",
Lines 164-172 class PatchAnalysisTask(object): a/Tools/Scripts/webkitpy/tool/bot/patchanalysistask.py_sec6
164	"--test",	202	"--test",
165	"--non-interactive",	203	"--non-interactive",
166	"--build-style=%s" % self._delegate.build_style(),	204	"--build-style=%s" % self._delegate.build_style(),
167	],	205	]
168	"Able to pass tests without patch",	206
169	"Unable to pass tests without patch (tree is red?)")	207	if hasattr(self._delegate, 'group'):
		208	args.append("--group=%s" % self._delegate.group())
		209
		210	return self._run_command(
		211	args,
		212	"Able to pass tests without patch",
		213	"Unable to pass tests without patch (tree is red?)")
170		214
171	def _land(self):	215	def _land(self):
172	# Unclear if this should pass --quiet or not. If --parent-command always does the reporting, then it should.	216	# Unclear if this should pass --quiet or not. If --parent-command always does the reporting, then it should.
Lines 210-278 class PatchAnalysisTask(object): a/Tools/Scripts/webkitpy/tool/bot/patchanalysistask.py_sec7
210	if self._test():	254	if self._test():
211	return True	255	return True
212		256
213	# Note: archive_last_test_results deletes the results directory, making these calls order-sensitve.	257	# FIXME: Abstract out the retry logic instead of casing on the test suite here.
214	# We could remove this dependency by building the test_results from the archive.	258	if hasattr(self._delegate, 'group') and self._delegate.group() == "jsc":
215	first_results = self._delegate.test_results()	259	first_results = self._delegate.test_results()
216	first_results_archive = self._delegate.archive_last_test_results(self._patch)	260	first_script_error = self._script_error
217	first_script_error = self._script_error	261	first_failure_status_id = self.failure_status_id
218	first_failure_status_id = self.failure_status_id	262	if first_results is None:
219		263	return False
220	if self._test() and not first_results.did_exceed_test_failure_limit():
221	# Only report flaky tests if we were successful at parsing results.json and archiving results.
222	if first_results and first_results_archive:
223	self._report_flaky_tests(first_results.failing_test_results(), first_results_archive)
224	return True
225		264
226	second_results = self._delegate.test_results()	265	if self._test():
227	second_results_archive = self._delegate.archive_last_test_results(self._patch)	266	return True
228	second_script_error = self._script_error	267	second_results = self._delegate.test_results()
229	second_failure_status_id = self.failure_status_id	268	second_script_error = self._script_error
		269	if second_results is None:
		270	return False
		271
		272	consistent_results = JSCTestResults.intersection(first_results, second_results)
230		273
231	if second_results.did_exceed_test_failure_limit() and first_results.did_exceed_test_failure_limit():
232	self._build_and_test_without_patch()	274	self._build_and_test_without_patch()
233	clean_tree_results = self._delegate.test_results()	275	clean_tree_results = self._delegate.test_results()
234		276	if clean_tree_results is None:
235	if (len(first_results.failing_tests()) - len(clean_tree_results.failing_tests())) <= 5:
236	return False	277	return False
237		278
		279	if consistent_results.is_subset(clean_tree_results):
		280	return True
		281
238	self.failure_status_id = first_failure_status_id	282	self.failure_status_id = first_failure_status_id
		283	return self.report_failure(None, consistent_results, first_script_error)
239		284
240	return self.report_failure(first_results_archive, first_results, first_script_error)	285	else:
		286	# Note: archive_last_test_results deletes the results directory, making these calls order-sensitve.
		287	# We could remove this dependency by building the test_results from the archive.
		288	first_results = self._delegate.test_results()
		289	first_results_archive = self._delegate.archive_last_test_results(self._patch)
		290	first_script_error = self._script_error
		291	first_failure_status_id = self.failure_status_id
241		292
242	if second_results.did_exceed_test_failure_limit():	293	if self._test() and not first_results.did_exceed_test_failure_limit():
243	self._should_defer_patch_or_throw(first_results.failing_test_results(), first_results_archive, first_script_error, first_failure_status_id)	294	# Only report flaky tests if we were successful at parsing results.json and archiving results.
244	return False	295	if first_results and first_results_archive:
		296	self._report_flaky_tests(first_results.failing_test_results(), first_results_archive)
		297	return True
245		298
246	if first_results.did_exceed_test_failure_limit():	299	second_results = self._delegate.test_results()
247	self._should_defer_patch_or_throw(second_results.failing_test_results(), second_results_archive, second_script_error, second_failure_status_id)	300	second_results_archive = self._delegate.archive_last_test_results(self._patch)
248	return False	301	second_script_error = self._script_error
		302	second_failure_status_id = self.failure_status_id
		303
		304	if second_results.did_exceed_test_failure_limit() and first_results.did_exceed_test_failure_limit():
		305	self._build_and_test_without_patch()
		306	clean_tree_results = self._delegate.test_results()
		307
		308	if (len(first_results.failing_tests()) - len(clean_tree_results.failing_tests())) <= 5:
		309	return False
		310
		311	self.failure_status_id = first_failure_status_id
		312
		313	return self.report_failure(first_results_archive, first_results, first_script_error)
249		314
250	if self._results_failed_different_tests(first_results, second_results):	315	if second_results.did_exceed_test_failure_limit():
251	first_failing_results_set = frozenset(first_results.failing_test_results())	316	self._should_defer_patch_or_throw(first_results.failing_test_results(), first_results_archive, first_script_error, first_failure_status_id)
252	second_failing_results_set = frozenset(second_results.failing_test_results())	317	return False
		318
		319	if first_results.did_exceed_test_failure_limit():
		320	self._should_defer_patch_or_throw(second_results.failing_test_results(), second_results_archive, second_script_error, second_failure_status_id)
		321	return False
253		322
254	tests_that_only_failed_first = first_failing_results_set.difference(second_failing_results_set)	323	if self._results_failed_different_tests(first_results, second_results):
255	self._report_flaky_tests(tests_that_only_failed_first, first_results_archive)	324	first_failing_results_set = frozenset(first_results.failing_test_results())
		325	second_failing_results_set = frozenset(second_results.failing_test_results())
256		326
257	tests_that_only_failed_second = second_failing_results_set.difference(first_failing_results_set)	327	tests_that_only_failed_first = first_failing_results_set.difference(second_failing_results_set)
258	self._report_flaky_tests(tests_that_only_failed_second, second_results_archive)	328	self._report_flaky_tests(tests_that_only_failed_first, first_results_archive)
259		329
260	tests_that_consistently_failed = first_failing_results_set.intersection(second_failing_results_set)	330	tests_that_only_failed_second = second_failing_results_set.difference(first_failing_results_set)
261	if tests_that_consistently_failed:	331	self._report_flaky_tests(tests_that_only_failed_second, second_results_archive)
262	if self._should_defer_patch_or_throw(tests_that_consistently_failed, first_results_archive, first_script_error, first_failure_status_id):
263	return False # Defer patch
264		332
265	# At this point we know that at least one test flaked, but no consistent failures	333	tests_that_consistently_failed = first_failing_results_set.intersection(second_failing_results_set)
266	# were introduced. This is a bit of a grey-zone.	334	if tests_that_consistently_failed:
267	return False # Defer patch	335	if self._should_defer_patch_or_throw(tests_that_consistently_failed, first_results_archive, first_script_error, first_failure_status_id):
		336	return False # Defer patch
268		337
269	if self._should_defer_patch_or_throw(first_results.failing_test_results(), first_results_archive, first_script_error, first_failure_status_id):	338	# At this point we know that at least one test flaked, but no consistent failures
270	return False # Defer patch	339	# were introduced. This is a bit of a grey-zone.
		340	return False # Defer patch
271		341
272	# At this point, we know that the first and second runs had the exact same failures,	342	if self._should_defer_patch_or_throw(first_results.failing_test_results(), first_results_archive, first_script_error, first_failure_status_id):
273	# and that those failures are all present on the clean tree, so we can say with certainty	343	return False # Defer patch
274	# that the patch is good.	344
275	return True	345	# At this point, we know that the first and second runs had the exact same failures,
		346	# and that those failures are all present on the clean tree, so we can say with certainty
		347	# that the patch is good.
		348	return True
276		349
277	def results_archive_from_patch_test_run(self, patch):	350	def results_archive_from_patch_test_run(self, patch):
278	assert(self._patch.id() == patch.id()) # PatchAnalysisTask is not currently re-useable.	351	assert(self._patch.id() == patch.id()) # PatchAnalysisTask is not currently re-useable.

Lines 1-5 a/Tools/Scripts/webkitpy/tool/commands/download.py_sec1
- a/Tools/Scripts/webkitpy/tool/commands/download.py -1 / +9 lines
1	# Copyright (c) 2009, 2011 Google Inc. All rights reserved.	1	# Copyright (c) 2009, 2011 Google Inc. All rights reserved.
2	# Copyright (c) 2009 Apple Inc. All rights reserved.	2	# Copyright (c) 2009, 2017 Apple Inc. All rights reserved.
3	#	3	#
4	# Redistribution and use in source and binary forms, with or without	4	# Redistribution and use in source and binary forms, with or without
5	# modification, are permitted provided that the following conditions are	5	# modification, are permitted provided that the following conditions are
Lines 87-92 class BuildAndTest(AbstractSequencedCommand): a/Tools/Scripts/webkitpy/tool/commands/download.py_sec2
87	]	87	]
88		88
89		89
		90	class CheckPatchRelevance(AbstractSequencedCommand):
		91	name = "check-patch-relevance"
		92	help_text = "Check if this patch needs to be tested"
		93	steps = [
		94	steps.CheckPatchRelevance,
		95	]
		96
		97
90	class Land(AbstractSequencedCommand):	98	class Land(AbstractSequencedCommand):
91	name = "land"	99	name = "land"
92	help_text = "Land the current working directory diff and updates the associated bug if any"	100	help_text = "Land the current working directory diff and updates the associated bug if any"



# Copyright (c) 2009 Google Inc. All rights reserved.
# Copyright (c) 2017 Apple Inc. All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are


import json
import logging
import os
from optparse import make_option

from webkitpy.common.config.committers import CommitterList

from webkitpy.common.system.executive import ScriptError
from webkitpy.tool.bot.earlywarningsystemtask import EarlyWarningSystemTask, EarlyWarningSystemTaskDelegate
from webkitpy.tool.bot.layouttestresultsreader import LayoutTestResultsReader
from webkitpy.tool.bot.jsctestresultsreader import JSCTestResultsReader
from webkitpy.tool.bot.patchanalysistask import UnableToApplyPatch, PatchIsNotValid, PatchIsNotApplicable
from webkitpy.tool.bot.queueengine import QueueEngine
from webkitpy.tool.commands.queues import AbstractReviewQueue



    def begin_work_queue(self):
        AbstractReviewQueue.begin_work_queue(self)

        if self.group() == "jsc":
            dir = os.path.dirname(os.path.dirname(os.path.dirname(self._port.results_directory())))
            self._test_results_reader = JSCTestResultsReader(self._tool, dir)
        else:
            self._test_results_reader = LayoutTestResultsReader(self._tool, self._port.results_directory(), self._log_directory())

    def _failing_tests_message(self, task, patch):
        results = task.results_from_patch_test_run(patch)

            tool.bugs.add_cc_to_bug(patch.bug_id(), self.watchers)
        tool.bugs.set_flag_on_attachment(patch.id(), "commit-queue", "-", message)

    # This exists for mocking
    def _create_task(self, patch):
        return EarlyWarningSystemTask(self, patch, self._options.run_tests)

    def review_patch(self, patch):
        task = self._create_task(patch)
        try:
            succeeded = task.run()
            if not succeeded:

        except UnableToApplyPatch, e:
            self._did_error(patch, "%s unable to apply patch." % self.name)
            return False
        except PatchIsNotApplicable, e:
            self._update_status("Patch not applicable to %s." % self.name, patch)
            self._release_work_item(patch)
            return False
        except ScriptError, e:
            self._post_reject_message_on_bug(self._tool, patch, task.failure_status_id, self._failing_tests_message(task, patch))
            results_archive = task.results_archive_from_patch_test_run(patch)

        return self._update_status(message, patch=patch, results_file=failure_log)

    def test_results(self):
        return self._test_results_reader.results()

    def archive_last_test_results(self, patch):
        return self._test_results_reader.archive(patch)

    def build_style(self):
        return self._build_style

    def group(self):
        return self._group

    def refetch_patch(self, patch):
        return self._tool.bugs.fetch_attachment(patch.id())



        classes = []
        for name, config in ewses.iteritems():
            classes.append(type(name.encode('utf-8').translate(None, ' -'), (cls,), {
                'name': config.get('name', config['port'] + '-ews'),
                'port_name': config['port'],
                'architecture': config.get('architecture', None),
                '_build_style': config.get('style', "release"),
                'watchers': config.get('watchers', []),
                'run_tests': config.get('runTests', cls.run_tests),
                '_group': config.get('group', None),
            }))
        return classes

Lines 1-4 a/Tools/Scripts/webkitpy/tool/commands/earlywarningsystem_unittest.py_sec1
- a/Tools/Scripts/webkitpy/tool/commands/earlywarningsystem_unittest.py -8 / +58 lines
1	# Copyright (C) 2009 Google Inc. All rights reserved.	1	# Copyright (C) 2009 Google Inc. All rights reserved.
		2	# Copyright (C) 2017 Apple Inc. All rights reserved.
2	#	3	#
3	# Redistribution and use in source and binary forms, with or without	4	# Redistribution and use in source and binary forms, with or without
4	# modification, are permitted provided that the following conditions are	5	# modification, are permitted provided that the following conditions are
Lines 29-34 a/Tools/Scripts/webkitpy/tool/commands/earlywarningsystem_unittest.py_sec2
29	from webkitpy.thirdparty.mock import Mock	30	from webkitpy.thirdparty.mock import Mock
30	from webkitpy.common.host import Host	31	from webkitpy.common.host import Host
31	from webkitpy.common.host_mock import MockHost	32	from webkitpy.common.host_mock import MockHost
		33	from webkitpy.common.net.jsctestresults import JSCTestResults
32	from webkitpy.common.net.layouttestresults import LayoutTestResults	34	from webkitpy.common.net.layouttestresults import LayoutTestResults
33	from webkitpy.common.system.outputcapture import OutputCapture	35	from webkitpy.common.system.outputcapture import OutputCapture
34	from webkitpy.layout_tests.models import test_results	36	from webkitpy.layout_tests.models import test_results
Lines 47-52 class AbstractEarlyWarningSystemTest(QueuesTest): a/Tools/Scripts/webkitpy/tool/commands/earlywarningsystem_unittest.py_sec3
47	class TestEWS(AbstractEarlyWarningSystem):	49	class TestEWS(AbstractEarlyWarningSystem):
48	port_name = "win" # Needs to be a port which port/factory understands.	50	port_name = "win" # Needs to be a port which port/factory understands.
49	_build_style = None	51	_build_style = None
		52	_group = None
50		53
51	ews = TestEWS()	54	ews = TestEWS()
52	ews.bind_to_tool(MockTool())	55	ews.bind_to_tool(MockTool())
Lines 60-102 class AbstractEarlyWarningSystemTest(QueuesTest): a/Tools/Scripts/webkitpy/tool/commands/earlywarningsystem_unittest.py_sec4
60	patch = ews._tool.bugs.fetch_attachment(10000)	63	patch = ews._tool.bugs.fetch_attachment(10000)
61	self.assertMultiLineEqual(ews._failing_tests_message(task, patch), "New failing tests:\nfoo.html\nbar.html")	64	self.assertMultiLineEqual(ews._failing_tests_message(task, patch), "New failing tests:\nfoo.html\nbar.html")
62		65
		66	def test_failing_jsc_tests_message(self):
		67	# Needed to define port_name, used in AbstractEarlyWarningSystem.__init__
		68	class TestEWS(AbstractEarlyWarningSystem):
		69	port_name = "mac" # Needs to be a port which port/factory understands.
		70	_build_style = None
		71	_group = "jsc"
		72
		73	ews = TestEWS()
		74	ews.bind_to_tool(MockTool())
		75	ews.host = MockHost()
		76	ews._options = MockOptions(port=None, confirm=False)
		77	OutputCapture().assert_outputs(self, ews.begin_work_queue, expected_logs=self._default_begin_work_queue_logs(ews.name))
		78	task = Mock()
		79	task.results_from_patch_test_run = lambda a: JSCTestResults(False, ["es6.yaml/es6/typed_arrays_Int8Array.js.default", "es6.yaml/es6/typed_arrays_Uint8Array.js.default"])
		80	patch = ews._tool.bugs.fetch_attachment(10000)
		81	self.assertMultiLineEqual(ews._failing_tests_message(task, patch), "New failing tests:\nes6.yaml/es6/typed_arrays_Int8Array.js.default\nes6.yaml/es6/typed_arrays_Uint8Array.js.default\napiTests")
		82
		83
		84	class MockEarlyWarningSystemTaskForInconclusiveResults(EarlyWarningSystemTask):
		85	def _test_patch(self):
		86	self._test()
		87	return False
		88
		89
		90	class MockAbstractEarlyWarningSystemForInconclusiveResults(AbstractEarlyWarningSystem):
		91	def test_results(self):
		92	return None
		93
		94	def _create_task(self, patch):
		95	task = MockEarlyWarningSystemTaskForInconclusiveResults(self, patch, self._options.run_tests)
		96	return task
		97
63		98
64	class EarlyWarningSystemTest(QueuesTest):	99	class EarlyWarningSystemTest(QueuesTest):
65	def _default_expected_logs(self, ews):	100	def _default_expected_logs(self, ews, conclusive):
66	string_replacements = {	101	string_replacements = {
67	"name": ews.name,	102	"name": ews.name,
68	"port": ews.port_name,	103	"port": ews.port_name,
69	"architecture": " --architecture=%s" % ews.architecture if ews.architecture else "",	104	"architecture": " --architecture=%s" % ews.architecture if ews.architecture else "",
70	"build_style": ews.build_style(),	105	"build_style": ews.build_style(),
		106	"group": ews.group(),
71	}	107	}
72	if ews.run_tests:	108	if ews.run_tests:
73	run_tests_line = "Running: webkit-patch --status-host=example.com build-and-test --no-clean --no-update --test --non-interactive --build-style=%(build_style)s --port=%(port)s%(architecture)s\n" % string_replacements	109	run_tests_line = "Running: webkit-patch --status-host=example.com build-and-test --no-clean --no-update --test --non-interactive --build-style=%(build_style)s --group=%(group)s --port=%(port)s%(architecture)s\n" % string_replacements
74	else:	110	else:
75	run_tests_line = ""	111	run_tests_line = ""
76	string_replacements['run_tests_line'] = run_tests_line	112	string_replacements['run_tests_line'] = run_tests_line
77		113
		114	if conclusive:
		115	result_lines = "MOCK: update_status: %(name)s Pass\nMOCK: release_work_item: %(name)s 10000\n" % string_replacements
		116	else:
		117	result_lines = "MOCK: release_lock: %(name)s 10000\n" % string_replacements
		118	string_replacements['result_lines'] = result_lines
		119
78	expected_logs = {	120	expected_logs = {
79	"begin_work_queue": self._default_begin_work_queue_logs(ews.name),	121	"begin_work_queue": self._default_begin_work_queue_logs(ews.name),
80	"process_work_item": """MOCK: update_status: %(name)s Started processing patch	122	"process_work_item": """MOCK: update_status: %(name)s Started processing patch
81	Running: webkit-patch --status-host=example.com clean --port=%(port)s%(architecture)s	123	Running: webkit-patch --status-host=example.com clean --port=%(port)s%(architecture)s
82	Running: webkit-patch --status-host=example.com update --port=%(port)s%(architecture)s	124	Running: webkit-patch --status-host=example.com update --port=%(port)s%(architecture)s
83	Running: webkit-patch --status-host=example.com apply-attachment --no-update --non-interactive 10000 --port=%(port)s%(architecture)s	125	Running: webkit-patch --status-host=example.com apply-attachment --no-update --non-interactive 10000 --port=%(port)s%(architecture)s
84	Running: webkit-patch --status-host=example.com build --no-clean --no-update --build-style=%(build_style)s --port=%(port)s%(architecture)s	126	Running: webkit-patch --status-host=example.com check-patch-relevance --group=%(group)s --port=%(port)s%(architecture)s
85	%(run_tests_line)sMOCK: update_status: %(name)s Pass	127	Running: webkit-patch --status-host=example.com build --no-clean --no-update --build-style=%(build_style)s --group=%(group)s --port=%(port)s%(architecture)s
86	MOCK: release_work_item: %(name)s 10000	128	%(run_tests_line)s%(result_lines)s""" % string_replacements,
87	""" % string_replacements,
88	"handle_unexpected_error": "Mock error message\n",	129	"handle_unexpected_error": "Mock error message\n",
89	"handle_script_error": "ScriptError error message\n\nMOCK output\n",	130	"handle_script_error": "ScriptError error message\n\nMOCK output\n",
90	}	131	}
91	return expected_logs	132	return expected_logs
92		133
93	def _test_ews(self, ews):	134	def _test_ews(self, ews, results_are_conclusive=True):
94	ews.bind_to_tool(MockTool())	135	ews.bind_to_tool(MockTool())
95	ews.host = MockHost()	136	ews.host = MockHost()
96	options = Mock()	137	options = Mock()
97	options.port = None	138	options.port = None
98	options.run_tests = ews.run_tests	139	options.run_tests = ews.run_tests
99	self.assert_queue_outputs(ews, expected_logs=self._default_expected_logs(ews), options=options)	140	self.assert_queue_outputs(ews, expected_logs=self._default_expected_logs(ews, results_are_conclusive), options=options)
100		141
101	def test_ewses(self):	142	def test_ewses(self):
102	classes = AbstractEarlyWarningSystem.load_ews_classes()	143	classes = AbstractEarlyWarningSystem.load_ews_classes()
Lines 104-106 MOCK: release_work_item: %(name)s 10000 a/Tools/Scripts/webkitpy/tool/commands/earlywarningsystem_unittest.py_sec5
104	self.maxDiff = None	145	self.maxDiff = None
105	for ews_class in classes:	146	for ews_class in classes:
106	self._test_ews(ews_class())	147	self._test_ews(ews_class())
		148
		149	# This must use the Mock EWS classes defined above
		150	def test_inconclusive_test_results(self):
		151	classes = MockAbstractEarlyWarningSystemForInconclusiveResults.load_ews_classes()
		152	self.assertTrue(classes)
		153	self.maxDiff = None
		154	test_classes = filter(lambda x: x.run_tests, classes)
		155	for ews_class in test_classes:
		156	self._test_ews(ews_class(), False)



# Copyright (C) 2010 Google Inc. All rights reserved.
# Copyright (C) 2017 Apple Inc. All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are

from webkitpy.tool.steps.applywatchlist import ApplyWatchList
from webkitpy.tool.steps.attachtobug import AttachToBug
from webkitpy.tool.steps.build import Build
from webkitpy.tool.steps.checkpatchrelevance import CheckPatchRelevance
from webkitpy.tool.steps.checkstyle import CheckStyle
from webkitpy.tool.steps.cleanworkingdirectory import CleanWorkingDirectory
from webkitpy.tool.steps.closebug import CloseBug



# Copyright (C) 2010 Google Inc. All rights reserved.
# Copyright (C) 2017 Apple Inc. All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are

            Options.build,
            Options.quiet,
            Options.build_style,
            Options.group,
        ]

    def build(self, build_style, group):
        environment = self._tool.copy_current_environment()
        environment.disable_gcc_smartquotes()
        environment.disable_jhbuild_VT100_output()
        env = environment.to_dictionary()

        if group == "jsc":
            build_command = self._tool.deprecated_port().build_jsc_command(build_style=build_style)
        else:
            build_command = self._tool.deprecated_port().build_webkit_command(build_style=build_style)

        if self._options.architecture:
            build_command += ['ARCHS=%s' % self._options.architecture]

        self._tool.executive.run_and_throw_if_fail(build_command, self._options.quiet,
            cwd=self._tool.scm().checkout_root, env=env)

    def run(self, state):
        if not self._options.build:
            return
        _log.info("Building WebKit")

        group = None
        if hasattr(self._options, 'group'):
            group = self._options.group

        if self._options.build_style == "both":
            self.build("debug", group)
            self.build("release", group)
        else:
            self.build(self._options.build_style, group)



# Copyright (C) 2017 Apple Inc. All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions
# are met:
# 1.  Redistributions of source code must retain the above copyright
#     notice, this list of conditions and the following disclaimer.
# 2.  Redistributions in binary form must reproduce the above copyright
#     notice, this list of conditions and the following disclaimer in the
#     documentation and/or other materials provided with the distribution.
#
# THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS'' AND
# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
# DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS BE LIABLE FOR
# ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
# SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

import logging
import re

from webkitpy.common.system.executive import ScriptError
from webkitpy.tool.steps.abstractstep import AbstractStep
from webkitpy.tool.steps.options import Options

_log = logging.getLogger(__name__)


class CheckPatchRelevance(AbstractStep):
    @classmethod
    def options(cls):
        return AbstractStep.options() + [
            Options.group,
        ]

    jsc_paths = [
        "JSTests/",
        "Source/JavaScriptCore/",
        "Source/WTF/"
        "Source/bmalloc/",
    ]

    def _does_contain_change_in_paths(self, files, patterns):
        for item in files:
            for pattern in patterns:
                if re.search(pattern, item, re.IGNORECASE):
                    return True

        return False

    def run(self, state):
        _log.info("Checking relevance of patch")

        group = None
        if hasattr(self._options, 'group'):
            group = self._options.group

        change_list = self._tool.scm().changed_files()

        if group == "jsc":
            relevant = self._does_contain_change_in_paths(change_list, self.jsc_paths)
        else:
            relevant = True

        if relevant is True:
            _log.info("This patch has relevant changes.")
            return True
        else:
            _log.info("This patch does not have relevant changes.")
            raise ScriptError(message="This patch does not have relevant changes.")



# Copyright (C) 2010 Google Inc. All rights reserved.
# Copyright (C) 2017 Apple Inc. All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are

    email = make_option("--email", action="store", type="string", dest="email", help="Email address to use in ChangeLogs.")
    force_clean = make_option("--force-clean", action="store_true", dest="force_clean", default=False, help="Clean working directory before applying patches (removes local changes and commits)")
    git_commit = make_option("-g", "--git-commit", action="store", dest="git_commit", help="Operate on a local commit. If a range, the commits are squashed into one. <ref>.... includes the working copy changes. UPSTREAM can be used for the upstream/tracking branch.")
    group = make_option("--group", action="store", dest="group", default=None, help="")
    local_commit = make_option("--local-commit", action="store_true", dest="local_commit", default=False, help="Make a local commit for each applied patch")
    non_interactive = make_option("--non-interactive", action="store_true", dest="non_interactive", default=False, help="Never prompt the user, fail as fast as possible.")
    obsolete_patches = make_option("--no-obsolete", action="store_false", dest="obsolete_patches", default=True, help="Do not obsolete old patches before posting this one.")

Lines 1-4 a/Tools/Scripts/webkitpy/tool/steps/runtests.py_sec1
- a/Tools/Scripts/webkitpy/tool/steps/runtests.py +10 lines
1	# Copyright (C) 2010 Google Inc. All rights reserved.	1	# Copyright (C) 2010 Google Inc. All rights reserved.
		2	# Copyright (C) 2017 Apple Inc. All rights reserved.
2	#	3	#
3	# Redistribution and use in source and binary forms, with or without	4	# Redistribution and use in source and binary forms, with or without
4	# modification, are permitted provided that the following conditions are	5	# modification, are permitted provided that the following conditions are
Lines 53-61 class RunTests(AbstractStep): a/Tools/Scripts/webkitpy/tool/steps/runtests.py_sec2
53	Options.iterate_on_new_tests,	54	Options.iterate_on_new_tests,
54	Options.non_interactive,	55	Options.non_interactive,
55	Options.quiet,	56	Options.quiet,
		57	Options.group,
56	]	58	]
57		59
58	def run(self, state):	60	def run(self, state):
		61	if hasattr(self._options, "group") and self._options.group == "jsc":
		62	self._run_javascriptcore_tests()
		63	return
		64
59	if self._options.iterate_on_new_tests:	65	if self._options.iterate_on_new_tests:
60	_log.info("Running run-webkit-tests on new tests")	66	_log.info("Running run-webkit-tests on new tests")
61	self._run_webkit_tests(self._options.iterate_on_new_tests)	67	self._run_webkit_tests(self._options.iterate_on_new_tests)
Lines 143-145 class RunTests(AbstractStep): a/Tools/Scripts/webkitpy/tool/steps/runtests.py_sec3
143	args.append("--iterations=%d" % iterate_on_new_tests)	149	args.append("--iterations=%d" % iterate_on_new_tests)
144		150
145	self._tool.executive.run_and_throw_if_fail(args, cwd=self._tool.scm().checkout_root)	151	self._tool.executive.run_and_throw_if_fail(args, cwd=self._tool.scm().checkout_root)
		152
		153	def _run_javascriptcore_tests(self):
		154	args = self._tool.deprecated_port().run_javascriptcore_tests_command(self._options.build_style)
		155	self._tool.executive.run_and_throw_if_fail(args, cwd=self._tool.scm().checkout_root)



# Copyright (C) 2010 Google Inc. All rights reserved.
# Copyright (C) 2017 Apple Inc. All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are


import unittest

from webkitpy.common.system.executive import ScriptError
from webkitpy.common.system.outputcapture import OutputCapture
from webkitpy.common.config.ports import DeprecatedPort
from webkitpy.tool.mocktool import MockOptions, MockTool

Running Perl unit tests
MOCK run_and_throw_if_fail: ['Tools/Scripts/test-webkitperl'], cwd=/mock-checkout
Running JavaScriptCore tests
MOCK run_and_throw_if_fail: ['Tools/Scripts/run-javascriptcore-tests', '--no-fail-fast', '--json-output=jsc_test_results.json'], cwd=/mock-checkout
Running bindings generation tests
MOCK run_and_throw_if_fail: ['Tools/Scripts/run-bindings-tests'], cwd=/mock-checkout
Running run-webkit-tests

Running Perl unit tests
MOCK run_and_throw_if_fail: ['Tools/Scripts/test-webkitperl'], cwd=/mock-checkout
Running JavaScriptCore tests
MOCK run_and_throw_if_fail: ['Tools/Scripts/run-javascriptcore-tests', '--no-fail-fast', '--json-output=jsc_test_results.json'], cwd=/mock-checkout
Running bindings generation tests
MOCK run_and_throw_if_fail: ['Tools/Scripts/run-bindings-tests'], cwd=/mock-checkout
Running run-webkit-tests
MOCK run_and_throw_if_fail: ['Tools/Scripts/run-webkit-tests', '--debug', '--quiet'], cwd=/mock-checkout
"""
        OutputCapture().assert_outputs(self, step.run, [{}], expected_logs=expected_logs)

    def test_runtests_jsc(self):
        mock_options = self._step_options()
        mock_options.non_interactive = False
        mock_options.build_style = "release"
        mock_options.group = "jsc"
        step = steps.RunTests(MockTool(log_executive=True), mock_options)
        tool = MockTool(log_executive=True)
        # FIXME: We shouldn't use a real port-object here, but there is too much to mock at the moment.
        tool._deprecated_port = DeprecatedPort()
        step = steps.RunTests(tool, mock_options)
        expected_logs = """MOCK run_and_throw_if_fail: ['Tools/Scripts/run-javascriptcore-tests', '--no-fail-fast', '--json-output=jsc_test_results.json'], cwd=/mock-checkout
"""
        OutputCapture().assert_outputs(self, step.run, [{}], expected_logs=expected_logs)

    def test_runtests_jsc_debug(self):
        mock_options = self._step_options()
        mock_options.non_interactive = False
        mock_options.build_style = "debug"
        mock_options.group = "jsc"
        tool = MockTool(log_executive=True)
        # FIXME: We shouldn't use a real port-object here, but there is too much to mock at the moment.
        tool._deprecated_port = DeprecatedPort()
        step = steps.RunTests(tool, mock_options)
        expected_logs = """MOCK run_and_throw_if_fail: ['Tools/Scripts/run-javascriptcore-tests', '--no-fail-fast', '--json-output=jsc_test_results.json', '--debug'], cwd=/mock-checkout
"""
        OutputCapture().assert_outputs(self, step.run, [{}], expected_logs=expected_logs)

    def test_build_jsc_debug(self):
        mock_options = self._step_options()
        mock_options.non_interactive = False
        mock_options.build_style = "debug"
        mock_options.build = True
        mock_options.architecture = True
        mock_options.group = "jsc"
        tool = MockTool(log_executive=True)
        # FIXME: We shouldn't use a real port-object here, but there is too much to mock at the moment.
        tool._deprecated_port = DeprecatedPort()
        step = steps.Build(tool, mock_options)
        expected_logs = """Building WebKit
MOCK run_and_throw_if_fail: ['Tools/Scripts/build-jsc', '--debug', 'ARCHS=True'], cwd=/mock-checkout, env={'LC_ALL': 'C', 'TERM': 'none', 'MOCK_ENVIRON_COPY': '1'}
"""
        OutputCapture().assert_outputs(self, step.run, [{}], expected_logs=expected_logs)

    def test_build_jsc(self):
        mock_options = self._step_options()
        mock_options.non_interactive = False
        mock_options.build_style = "release"
        mock_options.build = True
        mock_options.architecture = True
        mock_options.group = "jsc"
        tool = MockTool(log_executive=True)
        # FIXME: We shouldn't use a real port-object here, but there is too much to mock at the moment.
        tool._deprecated_port = DeprecatedPort()
        step = steps.Build(tool, mock_options)
        expected_logs = """Building WebKit
MOCK run_and_throw_if_fail: ['Tools/Scripts/build-jsc', '--release', 'ARCHS=True'], cwd=/mock-checkout, env={'LC_ALL': 'C', 'TERM': 'none', 'MOCK_ENVIRON_COPY': '1'}
"""
        OutputCapture().assert_outputs(self, step.run, [{}], expected_logs=expected_logs)

    def test_patch_relevant(self):
        self.maxDiff = None
        mock_options = self._step_options()
        tool = MockTool(log_executive=True)
        tool.scm()._mockChangedFiles = ["JSTests/MockFile1", "ChangeLog"]
        # FIXME: We shouldn't use a real port-object here, but there is too much to mock at the moment.
        tool._deprecated_port = DeprecatedPort()
        step = steps.CheckPatchRelevance(tool, mock_options)
        expected_logs = """Checking relevance of patch
This patch has relevant changes.
"""
        OutputCapture().assert_outputs(self, step.run, [{}], expected_logs=expected_logs)

    def test_patch_relevant_jsc(self):
        self.maxDiff = None
        mock_options = self._step_options()
        mock_options.group = "jsc"
        tool = MockTool(log_executive=True)
        tool.scm()._mockChangedFiles = ["JSTests/MockFile1", "ChangeLog"]
        # FIXME: We shouldn't use a real port-object here, but there is too much to mock at the moment.
        tool._deprecated_port = DeprecatedPort()
        step = steps.CheckPatchRelevance(tool, mock_options)
        expected_logs = """Checking relevance of patch
This patch has relevant changes.
"""
        OutputCapture().assert_outputs(self, step.run, [{}], expected_logs=expected_logs)

    def test_patch_not_relevant_jsc(self):
        self.maxDiff = None
        mock_options = self._step_options()
        mock_options.group = "jsc"
        tool = MockTool(log_executive=True)
        tool.scm()._mockChangedFiles = ["Tools/ChangeLog", "Tools/Scripts/webkitpy/tool/steps/steps_unittest.py"]
        # FIXME: We shouldn't use a real port-object here, but there is too much to mock at the moment.
        tool._deprecated_port = DeprecatedPort()
        step = steps.CheckPatchRelevance(tool, mock_options)
        expected_logs = """Checking relevance of patch
This patch does not have relevant changes.
"""
        with self.assertRaises(ScriptError):
            OutputCapture().assert_outputs(self, step.run, [{}], expected_logs=expected_logs)

Return to Bug 162458

Lines 1-3 a/Tools/ChangeLog_sec1
- a/Tools/ChangeLog +125 lines
		1	2017-02-08 Srinivasan Vijayaraghavan <svijayaraghavan@apple.com>
		2
		3	EWS should run JavaScriptCore tests
		4	https://bugs.webkit.org/show_bug.cgi?id=162458
		5
		6	Reviewed by NOBODY (OOPS!).
		7
		8	* QueueStatusServer/config/queues.py: Add jsc-ews queue
		9	* QueueStatusServer/handlers/statusbubble.py:
		10	(StatusBubble._should_show_bubble_for): Add logic to hide jsc-ews bubble if the patch does not touch jsc.
		11	* Scripts/webkitpy/common/checkout/scm/scm_mock.py:
		12	(MockSCM.__init__): Add _mockChangedFiles variable.
		13	(MockSCM.changed_files): Change to use _mockChangedFiles variables.
		14	* Scripts/webkitpy/common/config/ews.json: Add config info for JSC EWS.
		15	* Scripts/webkitpy/common/config/ports.py: Add support for JSC EWS in Mac Port.
		16	(DeprecatedPort.build_jsc_command): Added - command to build JSC only (quicker than building all of WebKit).
		17	(DeprecatedPort.run_javascriptcore_tests_command): Allow JSC EWS to only run JSC tests.
		18	(MacPort.run_webkit_tests_command): Check for JSC.
		19	* Scripts/webkitpy/common/config/ports_mock.py:
		20	(MockPort.run_javascriptcore_tests_command): Add build_style argument.
		21	* Scripts/webkitpy/common/config/ports_unittest.py:
		22	(DeprecatedPortTest.test_mac_port): Add unit tests for build-jsc shell command.
		23	* Scripts/webkitpy/common/net/jsctestresults.py: Added.
		24	(JSCTestResults): Added.
		25	(JSCTestResults.intersection): Return a JSCTestResults object with failures common to both input objects.
		26	(JSCTestResults.results_from_string): Use json library to parse results.
		27	(JSCTestResults.__init__): Initialize members.
		28	(JSCTestResults.equals): This enables unit testing.
		29	(JSCTestResults.is_subset): Checks if one set of failures is a subset of another.
		30	(JSCTestResults.stress_test_failures): Getter.
		31	(JSCTestResults.all_api_tests_passed): Getter.
		32	(JSCTestResults.all_passed): Getter.
		33	(JSCTestResults.failing_test_results): Getter. Some of these exist for interopability reasons.
		34	(JSCTestResults.test_results): Getter.
		35	(JSCTestResults.failing_tests): Getter.
		36	(JSCTestResults.did_exceed_test_failure_limit): Getter.
		37	* Scripts/webkitpy/common/net/jsctestresults_unittest.py: Added.
		38	(JSCTestResultsTest): Class to test JSCTestResults.
		39	(JSCTestResultsTest.test_results_from_string): Tests parsing.
		40	(JSCTestResultsTest.test_intersection_api_tests): Unit test for intersection() class method.
		41	(JSCTestResultsTest.test_intersection_stress_tests): Unit test for intersection() class method.
		42	(JSCTestResultsTest.test_intersection_general_case): Unit test for intersection() class method.
		43	* Scripts/webkitpy/tool/bot/earlywarningsystemtask.py:
		44	(EarlyWarningSystemTask.run): Add check_patch_relevance step.
		45	* Scripts/webkitpy/tool/bot/jscews_unittest.py: Added.
		46	(MockPatchAnalysisTask): Mocked-out version of PatchAnalysisTask that doesn't run commands.
		47	(MockPatchAnalysisTask.__init__): Sets attributes.
		48	(MockPatchAnalysisTask._test): Override to not run command.
		49	(MockPatchAnalysisTask._build_and_test_without_patch): Override to not run command.
		50	(MockPatchAnalysisTask.validate): Assume mocked patch is valid for purposes of testing retry logic.
		51	(MockPatchAnalysisTask.test_run_count): Specific to the mocked version, to test retry logic.
		52	(MockJSCEarlyWarningSystem): Mocked-out version of AbstarctEarlyWarningSystem so we can provide test results.
		53	(MockJSCEarlyWarningSystem.__init__): Sets attributes, also sets group to jsc.
		54	(MockJSCEarlyWarningSystem.test_results): Returns test results provided by us, instead of using a JSON reader.
		55	(JSCEarlyWarningSystemTest): Class to test retry logic in below situations.
		56	(JSCEarlyWarningSystemTest._create_task): Helper function to abstract out common code.
		57	(JSCEarlyWarningSystemTest._results_indicate_all_passed): False if input is None or has failures, else True.
		58	(JSCEarlyWarningSystemTest.test_success_case): Clean patch on clean tree.
		59	(JSCEarlyWarningSystemTest.test_test_failure): Failed patch on clean tree.
		60	(JSCEarlyWarningSystemTest.test_fix): Patch that fixes all tree redness.
		61	(JSCEarlyWarningSystemTest.test_ineffective_patch): Patch that has same failures as tree.
		62	(JSCEarlyWarningSystemTest.test_partially_effective_patch): Patch fixes some failures but adds no new failures.
		63	(JSCEarlyWarningSystemTest.test_different_test_failures_in_patch_and_tree): Patch has some failures not in tree.
		64	(JSCEarlyWarningSystemTest.test_first_results_could_not_be_read): Patch results not readable.
65	(JSCEarlyWarningSystemTest.test_second_results_could_not_be_read): Patch results not readable on second run.
66	(JSCEarlyWarningSystemTest.test_clean_results_could_not_be_read): Results from clean tree not readable.
67	(JSCEarlyWarningSystemTest.test_flaky_results_on_clean_tree_pass): Patch has one flake and no failures.
68	(JSCEarlyWarningSystemTest.test_flaky_results_on_clean_tree_pass_v2): Patch has one flake and no failures.
69	(JSCEarlyWarningSystemTest.test_flaky_results_on_clean_tree_failure): Patch has flakes and failed tests.
70	(JSCEarlyWarningSystemTest.test_flaky_results_on_red_tree_pass): Patch has same failures as tree, plus a flake.
71	* Scripts/webkitpy/tool/bot/jsctestresultsreader.py: Added.
72	(JSCTestResultsReader): Reads results file.
73	(JSCTestResultsReader.__init__): Sets attributes.
74	(JSCTestResultsReader._read_file_contents): Reads file.
75	(JSCTestResultsReader._create_jsc_test_results): Creates a JSCTestResults object.
76	(JSCTestResultsReader.results): Invokes _create_jsc_test_results to return a JSCTestResults object.
77	* Scripts/webkitpy/tool/bot/patchanalysistask.py:
78	(PatchIsNotApplicable): Exception for when patch doesn't have relevant changes.
79	(PatchAnalysisTask._check_patch_relevance): Added.
80	(PatchAnalysisTask._build): Check for JSC.
81	(PatchAnalysisTask._build_without_patch): Check for JSC.
82	(PatchAnalysisTask._test): Check for JSC.
83	(PatchAnalysisTask._build_and_test_without_patch): Check for JSC.
84	(PatchAnalysisTask._test_patch): Add retry logic for JSC.
85	* Scripts/webkitpy/tool/commands/download.py:
86	(CheckPatchRelevance): Add check-patch-relevance command.
87	* Scripts/webkitpy/tool/commands/earlywarningsystem.py:
88	(AbstractEarlyWarningSystem._create_task): Abstract out to enable mocking.
89	(AbstractEarlyWarningSystem.begin_work_queue): Use JSCTestResultsReader not LayoutTestResultsReader in JSC EWS.
90	(AbstractEarlyWarningSystem.review_patch): Handle PatchIsApplicable
91	(AbstractEarlyWarningSystem.test_results): _layout_test_results_reader -> _test_results_reader.
92	(AbstractEarlyWarningSystem.archive_last_test_results): _layout_test_results_reader -> _test_results_reader.
93	(AbstractEarlyWarningSystem.group): This attribute determines the type of EWS (eg. JSC).
94	(AbstractEarlyWarningSystem.load_ews_classes): Add _group, and make classes of type cls to enable mocking.
95	* Scripts/webkitpy/tool/commands/earlywarningsystem_unittest.py:
96	(AbstractEarlyWarningSystemTest.test_failing_tests_message.TestEWS): Add _group.
97	(AbstractEarlyWarningSystemTest.test_failing_jsc_tests_message.TestEWS): Added test for jsc failures message.
98	(AbstractEarlyWarningSystemTest): Add _group variable.
99	(EarlyWarningSystemTest._default_expected_logs): Add check-patch-relevance step, inconclusive logs, and group.
100	(EarlyWarningSystemTest._test_ews): Add logs_are_conclusive option to pass through to default_expected_logs().
101	(EarlyWarningSystemTest.test_inconclusive_test_results): Test not removing patch from queue if not conclusive.
102	(MockAbstractEarlyWarningSystem): Added.
103	(MockAbstractEarlyWarningSystem.test_results): Simulates inconclusive results.
104	(MockAbstractEarlyWarningSystem._create_task): Use MockEarlyWarningSystemTask (not EarlyWarningSystemTask).
105	(MockEarlyWarningSystemTask): Added.
106	(MockEarlyWarningSystemTask._test_patch): Simulates running tests but not getting a conclusive result.
107	* Scripts/webkitpy/tool/steps/__init__.py: Add CheckPatchRelevance import.
108	* Scripts/webkitpy/tool/steps/build.py:
109	(Build.options): Check for JSC.
110	(Build.build): Check for JSC.
111	(Build.run): Check for JSC.
112	* Scripts/webkitpy/tool/steps/checkpatchrelevance.py: Added.
113	(CheckPatchRelevance): Added.
114	(CheckPatchRelevance._does_contain_change_in_paths): Abstract function to perform regex matching.
115	(CheckPatchRelevance.run): Check if changed files in patch belong in certain folders.
116	* Scripts/webkitpy/tool/steps/options.py:
117	(Options): Add --group command line option.
118	* Scripts/webkitpy/tool/steps/runtests.py:
119	(RunTests.options): Add group.
120	(RunTests.run): Check for JSC.
121	(RunTests._run_webkit_tests): Check for JSC.
122	(RunTests): Add _group attribute.
123	(RunTests._run_javascriptcore_tests): New.
124	* Scripts/webkitpy/tool/steps/steps_unittest.py: Unit tests.
125
1	2017-02-03 Simon Fraser <simon.fraser@apple.com>	126	2017-02-03 Simon Fraser <simon.fraser@apple.com>
2		127
3	Correctly set the "inWindow" flag for TileControllers that aren't the page tiles, and clarify "usingTiledBacking" logic	128	Correctly set the "inWindow" flag for TileControllers that aren't the page tiles, and clarify "usingTiledBacking" logic

Lines 1-4 a/Tools/QueueStatusServer/handlers/statusbubble.py_sec1
- a/Tools/QueueStatusServer/handlers/statusbubble.py -3 / +12 lines
1	# Copyright (C) 2009 Google Inc. All rights reserved.	1	# Copyright (C) 2009 Google Inc. All rights reserved.
		2	# Copyright (C) 2017 Apple Inc. All rights reserved.
2	#	3	#
3	# Redistribution and use in source and binary forms, with or without	4	# Redistribution and use in source and binary forms, with or without
4	# modification, are permitted provided that the following conditions are	5	# modification, are permitted provided that the following conditions are
Lines 182-192 class StatusBubble(webapp.RequestHandler): a/Tools/QueueStatusServer/handlers/statusbubble.py_sec2
182	return bubble	183	return bubble
183		184
184	def _should_show_bubble_for(self, attachment, queue):	185	def _should_show_bubble_for(self, attachment, queue):
185	# Any pending queue is shown.	186	# Any pending queue is shown.
186	if attachment.position_in_queue(queue):	187	if attachment.position_in_queue(queue):
187	return True	188	return True
188	# EWS queues are also shown when complete.	189
189	return bool(queue.is_ews() and attachment.status_for_queue(queue))	190	# EWS queues are also shown when complete
		191	# But JSC EWS bubble is not shown if the patch is not relevant to JSC.
		192	if not queue.is_ews():
		193	return False
		194
		195	status = attachment.status_for_queue(queue)
		196	if queue.name() == "jsc-ews":
		197	return not status.message.startswith("Patch not applicable")
		198	return bool(status)
190		199
191	def _build_bubbles_for_attachment(self, attachment):	200	def _build_bubbles_for_attachment(self, attachment):
192	show_submit_to_ews = True	201	show_submit_to_ews = True

Lines 1-4 a/Tools/Scripts/webkitpy/common/checkout/scm/scm_mock.py_sec1
- a/Tools/Scripts/webkitpy/common/checkout/scm/scm_mock.py -1 / +3 lines
1	# Copyright (C) 2011 Google Inc. All rights reserved.	1	# Copyright (C) 2011 Google Inc. All rights reserved.
		2	# Copyright (C) 2017 Apple Inc. All rights reserved.
2	#	3	#
3	# Redistribution and use in source and binary forms, with or without	4	# Redistribution and use in source and binary forms, with or without
4	# modification, are permitted provided that the following conditions are	5	# modification, are permitted provided that the following conditions are
Lines 37-42 class MockSCM(object): a/Tools/Scripts/webkitpy/common/checkout/scm/scm_mock.py_sec2
37	self.added_paths = set()	38	self.added_paths = set()
38	self._filesystem = filesystem or MockFileSystem()	39	self._filesystem = filesystem or MockFileSystem()
39	self._executive = executive or MockExecutive()	40	self._executive = executive or MockExecutive()
		41	self._mockChangedFiles = ["MockFile1"]
40		42
41	def add(self, destination_path):	43	def add(self, destination_path):
42	self.add_list([destination_path])	44	self.add_list([destination_path])
Lines 71-77 class MockSCM(object): a/Tools/Scripts/webkitpy/common/checkout/scm/scm_mock.py_sec3
71	return self._filesystem.join(self.checkout_root, *comps)	73	return self._filesystem.join(self.checkout_root, *comps)
72		74
73	def changed_files(self, git_commit=None):	75	def changed_files(self, git_commit=None):
74	return ["MockFile1"]	76	return self._mockChangedFiles
75		77
76	def changed_files_for_revision(self, revision):	78	def changed_files_for_revision(self, revision):
77	return ["MockFile1"]	79	return ["MockFile1"]

Line 0 a/Tools/Scripts/webkitpy/common/net/jsctestresults.py_sec1
- a/Tools/Scripts/webkitpy/common/net/jsctestresults.py +86 lines
		1	# Copyright (C) 2017 Apple Inc. All rights reserved.
		2	#
		3	# Redistribution and use in source and binary forms, with or without
		4	# modification, are permitted provided that the following conditions
		5	# are met:
		6	# 1. Redistributions of source code must retain the above copyright
		7	# notice, this list of conditions and the following disclaimer.
		8	# 2. Redistributions in binary form must reproduce the above copyright
		9	# notice, this list of conditions and the following disclaimer in the
		10	# documentation and/or other materials provided with the distribution.
		11	#
		12	# THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS'' AND
		13	# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
		14	# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
		15	# DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS BE LIABLE FOR
		16	# ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
		17	# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
		18	# SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
		19	# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
		20	# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
		21	# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
		22
		23	import json
		24	import logging
		25
		26	_log = logging.getLogger(__name__)
		27
		28
		29	class JSCTestResults(object):
		30	@classmethod
		31	def intersection(cls, first, second):
		32	intersection_api_tests_passed = first._all_api_tests_passed or second._all_api_tests_passed
		33	intersection_stress_test_failures = list(set(first._stress_test_failures) & set(second._stress_test_failures))
		34	return cls(intersection_api_tests_passed, intersection_stress_test_failures)
		35
		36	@classmethod
		37	def results_from_string(cls, string):
		38	if not string:
		39	return None
		40
		41	try:
		42	parsed_results = json.loads(string)
		43	except ValueError:
		44	return None
		45
		46	if 'allApiTestsPassed' not in parsed_results or 'stressTestFailures' not in parsed_results:
		47	return None
		48
		49	return cls(parsed_results['allApiTestsPassed'], parsed_results['stressTestFailures'])
		50
		51	def __init__(self, all_api_tests_passed, stress_test_failures):
		52	self._all_api_tests_passed = all_api_tests_passed
		53	self._stress_test_failures = stress_test_failures
		54
		55	self._test_results = stress_test_failures[:]
		56	if not self._all_api_tests_passed:
		57	self._test_results.append("apiTests")
		58
		59	def equals(self, other):
		60	return self._all_api_tests_passed == other._all_api_tests_passed and \
		61	set(self._stress_test_failures) == set(other._stress_test_failures)
		62
		63	def is_subset(self, other):
		64	return set(self.test_results()) <= set(other.test_results())
65
66	def stress_test_failures(self):
67	return self._stress_test_failures
68
69	def all_api_tests_passed(self):
70	return self._all_api_tests_passed
71
72	def all_passed(self):
73	return self._all_api_tests_passed and not self._stress_test_failures
74
75	def failing_test_results(self):
76	return self.stress_test_failures()
77
78	def test_results(self):
79	return self._test_results
80
81	def failing_tests(self):
82	return self._test_results
83
84	# No defined failure limit for JSC.
85	def did_exceed_test_failure_limit(self):
86	return False

Line 0 a/Tools/Scripts/webkitpy/tool/bot/jscews_unittest.py_sec1
- a/Tools/Scripts/webkitpy/tool/bot/jscews_unittest.py +219 lines
		1	# Copyright (C) 2017 Apple Inc. All rights reserved.
		2	#
		3	# Redistribution and use in source and binary forms, with or without
		4	# modification, are permitted provided that the following conditions
		5	# are met:
		6	# 1. Redistributions of source code must retain the above copyright
		7	# notice, this list of conditions and the following disclaimer.
		8	# 2. Redistributions in binary form must reproduce the above copyright
		9	# notice, this list of conditions and the following disclaimer in the
		10	# documentation and/or other materials provided with the distribution.
		11	#
		12	# THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS'' AND
		13	# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
		14	# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
		15	# DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS BE LIABLE FOR
		16	# ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
		17	# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
		18	# SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
		19	# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
		20	# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
		21	# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
		22
		23	import logging
		24	import unittest
		25
		26	from webkitpy.common.net.jsctestresults import JSCTestResults
		27	from webkitpy.common.system.executive import ScriptError
		28	from webkitpy.tool.bot.patchanalysistask import *
		29	from webkitpy.tool.commands.earlywarningsystem import AbstractEarlyWarningSystem
		30	from webkitpy.tool.mocktool import MockTool
		31
		32	_log = logging.getLogger(__name__)
		33
		34
		35	class MockPatchAnalysisTask(PatchAnalysisTask):
		36	def __init__(self, delegate, patch, patches_passed_all_tests):
		37	self._delegate = delegate
		38	self._patch = patch
		39	self._script_error = None
		40	self._results_archive_from_patch_test_run = None
		41	self._results_from_patch_test_run = None
		42	self.error = None
		43	self._patches_passed_all_tests = patches_passed_all_tests
		44	self._test_run_count = 0
		45	self.failure_status_id = 42
		46
		47	def _test(self):
		48	self._test_run_count += 1
		49
		50	if self._patches_passed_all_tests.pop() == True:
		51	return True
		52	else:
		53	self._script_error = ScriptError("Regression test")
		54	return False
		55
		56	def _build_and_test_without_patch(self):
		57	self._test_run_count += 1
		58	return True
		59
		60	def validate(self):
		61	return True
		62
		63	def test_run_count(self):
		64	return self._test_run_count
65
66
67	# This is the delegate to be used with MockPatchAnalysisTask, above.
68	class MockJSCEarlyWarningSystem(AbstractEarlyWarningSystem):
69	def __init__(self, first_test_results, second_test_results, clean_test_results):
70	AbstractEarlyWarningSystem.__init__(self)
71	self._group = "jsc"
72	self._results_in_order = [clean_test_results, second_test_results, first_test_results]
73
74	def test_results(self):
75	return self._results_in_order.pop()
76
77
78	class JSCEarlyWarningSystemTest(unittest.TestCase):
79	def _results_indicate_all_passed(self, results):
80	if results == None:
81	return False
82	return results.all_passed()
83
84	def _create_task(self, first_test_results, second_test_results, clean_test_results):
85	queue = MockJSCEarlyWarningSystem(first_test_results, second_test_results, clean_test_results)
86	tool = MockTool(log_executive=True)
87	patch = tool.bugs.fetch_attachment(10000)
88	patches_passed_all_tests = map(self._results_indicate_all_passed, [second_test_results, first_test_results])
89	return MockPatchAnalysisTask(queue, patch, patches_passed_all_tests)
90
91	def test_success_case(self):
92	first_test_results = JSCTestResults(True, [])
93	second_test_results = JSCTestResults(True, [])
94	clean_test_results = JSCTestResults(True, [])
95	task = self._create_task(first_test_results, second_test_results, clean_test_results)
96
97	return_value = task._test_patch()
98	self.assertEqual(task.test_run_count(), 1)
99	self.assertEqual(return_value, True)
100
101	def test_test_failure(self):
102	first_test_results = JSCTestResults(True, ['Fail.js'])
103	second_test_results = JSCTestResults(True, ['Fail.js'])
104	clean_test_results = JSCTestResults(True, [])
105	task = self._create_task(first_test_results, second_test_results, clean_test_results)
106
107	with self.assertRaises(ScriptError):
108	return_value = task._test_patch()
109	self.assertEqual(task.test_run_count(), 3)
110
111	def test_fix(self):
112	first_test_results = JSCTestResults(True, [])
113	second_test_results = JSCTestResults(True, [])
114	clean_test_results = JSCTestResults(True, ['Fail.js'])
115	task = self._create_task(first_test_results, second_test_results, clean_test_results)
116
117	return_value = task._test_patch()
118	self.assertEqual(task.test_run_count(), 1)
119	self.assertEqual(return_value, True)
120
121	def test_ineffective_patch(self):
122	first_test_results = JSCTestResults(False, ['failure1.js', 'failure2.js'])
123	second_test_results = JSCTestResults(False, ['failure1.js', 'failure2.js'])
124	clean_test_results = JSCTestResults(False, ['failure1.js', 'failure2.js'])
125	task = self._create_task(first_test_results, second_test_results, clean_test_results)
126
127	return_value = task._test_patch()
128	self.assertEqual(task.test_run_count(), 3)
129	self.assertEqual(return_value, True)
130
131	def test_partially_effective_patch(self):
132	first_test_results = JSCTestResults(True, ['failure2.js'])
133	second_test_results = JSCTestResults(True, ['failure2.js'])
134	clean_test_results = JSCTestResults(False, ['failure1.js', 'failure2.js'])
135	task = self._create_task(first_test_results, second_test_results, clean_test_results)
136
137	return_value = task._test_patch()
138	self.assertEqual(task.test_run_count(), 3)
139	self.assertEqual(return_value, True)
140
141	def test_different_test_failures_in_patch_and_tree(self):
142	first_test_results = JSCTestResults(False, [])
143	second_test_results = JSCTestResults(False, [])
144	clean_test_results = JSCTestResults(True, ['failure1.js', 'failure2.js'])
145	task = self._create_task(first_test_results, second_test_results, clean_test_results)
146
147	with self.assertRaises(ScriptError):
148	return_value = task._test_patch()
149	self.assertEqual(task.test_run_count(), 3)
150
151	def test_first_results_could_not_be_read(self):
152	first_test_results = None
153	second_test_results = JSCTestResults(True, [])
154	clean_test_results = JSCTestResults(True, [])
155	task = self._create_task(first_test_results, second_test_results, clean_test_results)
156
157	return_value = task._test_patch()
158	self.assertEqual(task.test_run_count(), 1)
159	self.assertEqual(return_value, False)
160
161	def test_second_results_could_not_be_read(self):
162	first_test_results = JSCTestResults(False, ['failure1.js', 'failure2.js'])
163	second_test_results = None
164	clean_test_results = JSCTestResults(True, [])
165	task = self._create_task(first_test_results, second_test_results, clean_test_results)
166
167	return_value = task._test_patch()
168	self.assertEqual(task.test_run_count(), 2)
169	self.assertEqual(return_value, False)
170
171	def test_clean_results_could_not_be_read(self):
172	first_test_results = JSCTestResults(True, ['failure2.js'])
173	second_test_results = JSCTestResults(True, ['failure2.js'])
174	clean_test_results = None
175	task = self._create_task(first_test_results, second_test_results, clean_test_results)
176
177	return_value = task._test_patch()
178	self.assertEqual(task.test_run_count(), 3)
179	self.assertEqual(return_value, False)
180
181	def test_flaky_results_on_clean_tree_pass(self):
182	first_test_results = JSCTestResults(True, ['failure2.js'])
183	second_test_results = JSCTestResults(True, [])
184	clean_test_results = JSCTestResults(True, [])
185	task = self._create_task(first_test_results, second_test_results, clean_test_results)
186
187	return_value = task._test_patch()
188	self.assertEqual(return_value, True)
189	self.assertEqual(task.test_run_count(), 2)
190
191	def test_flaky_results_on_clean_tree_pass_v2(self):
192	first_test_results = JSCTestResults(True, [])
193	second_test_results = JSCTestResults(True, ['failure2.js'])
194	clean_test_results = JSCTestResults(True, [])
195	task = self._create_task(first_test_results, second_test_results, clean_test_results)
196
197	return_value = task._test_patch()
198	self.assertEqual(return_value, True)
199	self.assertEqual(task.test_run_count(), 1)
200
201	def test_flaky_results_on_clean_tree_failure(self):
202	first_test_results = JSCTestResults(False, ['failure1.js', 'failure2.js'])
203	second_test_results = JSCTestResults(True, ['failure2.js'])
204	clean_test_results = JSCTestResults(True, [])
205	task = self._create_task(first_test_results, second_test_results, clean_test_results)
206
207	with self.assertRaises(ScriptError):
208	return_value = task._test_patch()
209	self.assertEqual(task.test_run_count(), 3)
210
211	def test_flaky_results_on_red_tree_pass(self):
212	first_test_results = JSCTestResults(True, ['failure1.js'])
213	second_test_results = JSCTestResults(True, ['failure1.js', 'failure2.js'])
214	clean_test_results = JSCTestResults(True, ['failure1.js'])
215	task = self._create_task(first_test_results, second_test_results, clean_test_results)
216
217	return_value = task._test_patch()
218	self.assertEqual(return_value, True)
219	self.assertEqual(task.test_run_count(), 3)

Line 0 a/Tools/Scripts/webkitpy/tool/bot/jsctestresultsreader.py_sec1
- a/Tools/Scripts/webkitpy/tool/bot/jsctestresultsreader.py +47 lines
	1	# Copyright (C) 2017 Apple Inc. All rights reserved.
	2	#
	3	# Redistribution and use in source and binary forms, with or without
	4	# modification, are permitted provided that the following conditions
	5	# are met:
	6	# 1. Redistributions of source code must retain the above copyright
	7	# notice, this list of conditions and the following disclaimer.
	8	# 2. Redistributions in binary form must reproduce the above copyright
	9	# notice, this list of conditions and the following disclaimer in the
	10	# documentation and/or other materials provided with the distribution.
	11	#
	12	# THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS'' AND
	13	# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
	14	# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
	15	# DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS BE LIABLE FOR
	16	# ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
	17	# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
	18	# SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
	19	# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
	20	# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
	21	# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
	22
	23	import logging
	24
	25	from webkitpy.common.net.jsctestresults import JSCTestResults
	26
	27	_log = logging.getLogger(__name__)
	28
	29
	30	class JSCTestResultsReader(object):
	31	def __init__(self, host, results_directory):
	32	self._host = host
	33	self._results_directory = results_directory
	34
	35	def _read_file_contents(self, path):
	36	try:
	37	return self._host.filesystem.read_text_file(path)
	38	except (IOError, KeyError): # File does not exist or can't be read.
	39	return None
	40
	41	def _create_jsc_test_results(self):
	42	results_path = self._host.filesystem.join(self._results_directory, "jsc_test_results.json")
	43	contents = self._read_file_contents(results_path)
	44	return JSCTestResults.results_from_string(contents)
	45
	46	def results(self):
	47	return self._create_jsc_test_results()

Lines 1-4 a/Tools/Scripts/webkitpy/tool/steps/__init__.py_sec1
- a/Tools/Scripts/webkitpy/tool/steps/__init__.py +2 lines
1	# Copyright (C) 2010 Google Inc. All rights reserved.	1	# Copyright (C) 2010 Google Inc. All rights reserved.
		2	# Copyright (C) 2017 Apple Inc. All rights reserved.
2	#	3	#
3	# Redistribution and use in source and binary forms, with or without	4	# Redistribution and use in source and binary forms, with or without
4	# modification, are permitted provided that the following conditions are	5	# modification, are permitted provided that the following conditions are
Lines 33-38 from webkitpy.tool.steps.applypatchwithlocalcommit import ApplyPatchWithLocalCom a/Tools/Scripts/webkitpy/tool/steps/__init__.py_sec2
33	from webkitpy.tool.steps.applywatchlist import ApplyWatchList	34	from webkitpy.tool.steps.applywatchlist import ApplyWatchList
34	from webkitpy.tool.steps.attachtobug import AttachToBug	35	from webkitpy.tool.steps.attachtobug import AttachToBug
35	from webkitpy.tool.steps.build import Build	36	from webkitpy.tool.steps.build import Build
		37	from webkitpy.tool.steps.checkpatchrelevance import CheckPatchRelevance
36	from webkitpy.tool.steps.checkstyle import CheckStyle	38	from webkitpy.tool.steps.checkstyle import CheckStyle
37	from webkitpy.tool.steps.cleanworkingdirectory import CleanWorkingDirectory	39	from webkitpy.tool.steps.cleanworkingdirectory import CleanWorkingDirectory
38	from webkitpy.tool.steps.closebug import CloseBug	40	from webkitpy.tool.steps.closebug import CloseBug

Lines 1-4 a/Tools/Scripts/webkitpy/tool/steps/build.py_sec1
- a/Tools/Scripts/webkitpy/tool/steps/build.py -7 / +17 lines
1	# Copyright (C) 2010 Google Inc. All rights reserved.	1	# Copyright (C) 2010 Google Inc. All rights reserved.
		2	# Copyright (C) 2017 Apple Inc. All rights reserved.
2	#	3	#
3	# Redistribution and use in source and binary forms, with or without	4	# Redistribution and use in source and binary forms, with or without
4	# modification, are permitted provided that the following conditions are	5	# modification, are permitted provided that the following conditions are
Lines 41-68 class Build(AbstractStep): a/Tools/Scripts/webkitpy/tool/steps/build.py_sec2
41	Options.build,	42	Options.build,
42	Options.quiet,	43	Options.quiet,
43	Options.build_style,	44	Options.build_style,
		45	Options.group,
44	]	46	]
45		47
46	def build(self, build_style):	48	def build(self, build_style, group):
47	environment = self._tool.copy_current_environment()	49	environment = self._tool.copy_current_environment()
48	environment.disable_gcc_smartquotes()	50	environment.disable_gcc_smartquotes()
49	environment.disable_jhbuild_VT100_output()	51	environment.disable_jhbuild_VT100_output()
50	env = environment.to_dictionary()	52	env = environment.to_dictionary()
51		53
52	build_webkit_command = self._tool.deprecated_port().build_webkit_command(build_style=build_style)	54	if group == "jsc":
		55	build_command = self._tool.deprecated_port().build_jsc_command(build_style=build_style)
		56	else:
		57	build_command = self._tool.deprecated_port().build_webkit_command(build_style=build_style)
53		58
54	if self._options.architecture:	59	if self._options.architecture:
55	build_webkit_command += ['ARCHS=%s' % self._options.architecture]	60	build_command += ['ARCHS=%s' % self._options.architecture]
56		61
57	self._tool.executive.run_and_throw_if_fail(build_webkit_command, self._options.quiet,	62	self._tool.executive.run_and_throw_if_fail(build_command, self._options.quiet,
58	cwd=self._tool.scm().checkout_root, env=env)	63	cwd=self._tool.scm().checkout_root, env=env)
59		64
60	def run(self, state):	65	def run(self, state):
61	if not self._options.build:	66	if not self._options.build:
62	return	67	return
63	_log.info("Building WebKit")	68	_log.info("Building WebKit")
		69
		70	group = None
		71	if hasattr(self._options, 'group'):
		72	group = self._options.group
		73
64	if self._options.build_style == "both":	74	if self._options.build_style == "both":
65	self.build("debug")	75	self.build("debug", group)
66	self.build("release")	76	self.build("release", group)
67	else:	77	else:
68	self.build(self._options.build_style)	78	self.build(self._options.build_style, group)

Line 0 a/Tools/Scripts/webkitpy/tool/steps/checkpatchrelevance.py_sec1
- a/Tools/Scripts/webkitpy/tool/steps/checkpatchrelevance.py +74 lines
		1	# Copyright (C) 2017 Apple Inc. All rights reserved.
		2	#
		3	# Redistribution and use in source and binary forms, with or without
		4	# modification, are permitted provided that the following conditions
		5	# are met:
		6	# 1. Redistributions of source code must retain the above copyright
		7	# notice, this list of conditions and the following disclaimer.
		8	# 2. Redistributions in binary form must reproduce the above copyright
		9	# notice, this list of conditions and the following disclaimer in the
		10	# documentation and/or other materials provided with the distribution.
		11	#
		12	# THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS'' AND
		13	# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
		14	# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
		15	# DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS BE LIABLE FOR
		16	# ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
		17	# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
		18	# SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
		19	# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
		20	# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
		21	# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
		22
		23	import logging
		24	import re
		25
		26	from webkitpy.common.system.executive import ScriptError
		27	from webkitpy.tool.steps.abstractstep import AbstractStep
		28	from webkitpy.tool.steps.options import Options
		29
		30	_log = logging.getLogger(__name__)
		31
		32
		33	class CheckPatchRelevance(AbstractStep):
		34	@classmethod
		35	def options(cls):
		36	return AbstractStep.options() + [
		37	Options.group,
		38	]
		39
		40	jsc_paths = [
		41	"JSTests/",
		42	"Source/JavaScriptCore/",
		43	"Source/WTF/"
		44	"Source/bmalloc/",
		45	]
		46
		47	def _does_contain_change_in_paths(self, files, patterns):
		48	for item in files:
		49	for pattern in patterns:
		50	if re.search(pattern, item, re.IGNORECASE):
		51	return True
		52
		53	return False
		54
		55	def run(self, state):
		56	_log.info("Checking relevance of patch")
		57
		58	group = None
		59	if hasattr(self._options, 'group'):
		60	group = self._options.group
		61
		62	change_list = self._tool.scm().changed_files()
		63
		64	if group == "jsc":
65	relevant = self._does_contain_change_in_paths(change_list, self.jsc_paths)
66	else:
67	relevant = True
68
69	if relevant is True:
70	_log.info("This patch has relevant changes.")
71	return True
72	else:
73	_log.info("This patch does not have relevant changes.")
74	raise ScriptError(message="This patch does not have relevant changes.")

Lines 1-4 a/Tools/Scripts/webkitpy/tool/steps/options.py_sec1
- a/Tools/Scripts/webkitpy/tool/steps/options.py +2 lines
1	# Copyright (C) 2010 Google Inc. All rights reserved.	1	# Copyright (C) 2010 Google Inc. All rights reserved.
		2	# Copyright (C) 2017 Apple Inc. All rights reserved.
2	#	3	#
3	# Redistribution and use in source and binary forms, with or without	4	# Redistribution and use in source and binary forms, with or without
4	# modification, are permitted provided that the following conditions are	5	# modification, are permitted provided that the following conditions are
Lines 46-51 class Options(object): a/Tools/Scripts/webkitpy/tool/steps/options.py_sec2
46	email = make_option("--email", action="store", type="string", dest="email", help="Email address to use in ChangeLogs.")	47	email = make_option("--email", action="store", type="string", dest="email", help="Email address to use in ChangeLogs.")
47	force_clean = make_option("--force-clean", action="store_true", dest="force_clean", default=False, help="Clean working directory before applying patches (removes local changes and commits)")	48	force_clean = make_option("--force-clean", action="store_true", dest="force_clean", default=False, help="Clean working directory before applying patches (removes local changes and commits)")
48	git_commit = make_option("-g", "--git-commit", action="store", dest="git_commit", help="Operate on a local commit. If a range, the commits are squashed into one. <ref>.... includes the working copy changes. UPSTREAM can be used for the upstream/tracking branch.")	49	git_commit = make_option("-g", "--git-commit", action="store", dest="git_commit", help="Operate on a local commit. If a range, the commits are squashed into one. <ref>.... includes the working copy changes. UPSTREAM can be used for the upstream/tracking branch.")
		50	group = make_option("--group", action="store", dest="group", default=None, help="")
49	local_commit = make_option("--local-commit", action="store_true", dest="local_commit", default=False, help="Make a local commit for each applied patch")	51	local_commit = make_option("--local-commit", action="store_true", dest="local_commit", default=False, help="Make a local commit for each applied patch")
50	non_interactive = make_option("--non-interactive", action="store_true", dest="non_interactive", default=False, help="Never prompt the user, fail as fast as possible.")	52	non_interactive = make_option("--non-interactive", action="store_true", dest="non_interactive", default=False, help="Never prompt the user, fail as fast as possible.")
51	obsolete_patches = make_option("--no-obsolete", action="store_false", dest="obsolete_patches", default=True, help="Do not obsolete old patches before posting this one.")	53	obsolete_patches = make_option("--no-obsolete", action="store_false", dest="obsolete_patches", default=True, help="Do not obsolete old patches before posting this one.")

Lines 1-4 a/Tools/Scripts/webkitpy/tool/steps/steps_unittest.py_sec1
- a/Tools/Scripts/webkitpy/tool/steps/steps_unittest.py -2 / +105 lines
1	# Copyright (C) 2010 Google Inc. All rights reserved.	1	# Copyright (C) 2010 Google Inc. All rights reserved.
		2	# Copyright (C) 2017 Apple Inc. All rights reserved.
2	#	3	#
3	# Redistribution and use in source and binary forms, with or without	4	# Redistribution and use in source and binary forms, with or without
4	# modification, are permitted provided that the following conditions are	5	# modification, are permitted provided that the following conditions are
Lines 28-33 a/Tools/Scripts/webkitpy/tool/steps/steps_unittest.py_sec2
28		29
29	import unittest	30	import unittest
30		31
		32	from webkitpy.common.system.executive import ScriptError
31	from webkitpy.common.system.outputcapture import OutputCapture	33	from webkitpy.common.system.outputcapture import OutputCapture
32	from webkitpy.common.config.ports import DeprecatedPort	34	from webkitpy.common.config.ports import DeprecatedPort
33	from webkitpy.tool.mocktool import MockOptions, MockTool	35	from webkitpy.tool.mocktool import MockOptions, MockTool
Lines 111-117 MOCK run_and_throw_if_fail: ['Tools/Scripts/test-webkitpy'], cwd=/mock-checkout a/Tools/Scripts/webkitpy/tool/steps/steps_unittest.py_sec3
111	Running Perl unit tests	113	Running Perl unit tests
112	MOCK run_and_throw_if_fail: ['Tools/Scripts/test-webkitperl'], cwd=/mock-checkout	114	MOCK run_and_throw_if_fail: ['Tools/Scripts/test-webkitperl'], cwd=/mock-checkout
113	Running JavaScriptCore tests	115	Running JavaScriptCore tests
114	MOCK run_and_throw_if_fail: ['Tools/Scripts/run-javascriptcore-tests'], cwd=/mock-checkout	116	MOCK run_and_throw_if_fail: ['Tools/Scripts/run-javascriptcore-tests', '--no-fail-fast', '--json-output=jsc_test_results.json'], cwd=/mock-checkout
115	Running bindings generation tests	117	Running bindings generation tests
116	MOCK run_and_throw_if_fail: ['Tools/Scripts/run-bindings-tests'], cwd=/mock-checkout	118	MOCK run_and_throw_if_fail: ['Tools/Scripts/run-bindings-tests'], cwd=/mock-checkout
117	Running run-webkit-tests	119	Running run-webkit-tests
Lines 133-142 MOCK run_and_throw_if_fail: ['Tools/Scripts/test-webkitpy'], cwd=/mock-checkout a/Tools/Scripts/webkitpy/tool/steps/steps_unittest.py_sec4
133	Running Perl unit tests	135	Running Perl unit tests
134	MOCK run_and_throw_if_fail: ['Tools/Scripts/test-webkitperl'], cwd=/mock-checkout	136	MOCK run_and_throw_if_fail: ['Tools/Scripts/test-webkitperl'], cwd=/mock-checkout
135	Running JavaScriptCore tests	137	Running JavaScriptCore tests
136	MOCK run_and_throw_if_fail: ['Tools/Scripts/run-javascriptcore-tests'], cwd=/mock-checkout	138	MOCK run_and_throw_if_fail: ['Tools/Scripts/run-javascriptcore-tests', '--no-fail-fast', '--json-output=jsc_test_results.json'], cwd=/mock-checkout
137	Running bindings generation tests	139	Running bindings generation tests
138	MOCK run_and_throw_if_fail: ['Tools/Scripts/run-bindings-tests'], cwd=/mock-checkout	140	MOCK run_and_throw_if_fail: ['Tools/Scripts/run-bindings-tests'], cwd=/mock-checkout
139	Running run-webkit-tests	141	Running run-webkit-tests
140	MOCK run_and_throw_if_fail: ['Tools/Scripts/run-webkit-tests', '--debug', '--quiet'], cwd=/mock-checkout	142	MOCK run_and_throw_if_fail: ['Tools/Scripts/run-webkit-tests', '--debug', '--quiet'], cwd=/mock-checkout
141	"""	143	"""
142	OutputCapture().assert_outputs(self, step.run, [{}], expected_logs=expected_logs)	144	OutputCapture().assert_outputs(self, step.run, [{}], expected_logs=expected_logs)
		145
		146	def test_runtests_jsc(self):
		147	mock_options = self._step_options()
		148	mock_options.non_interactive = False
		149	mock_options.build_style = "release"
		150	mock_options.group = "jsc"
		151	step = steps.RunTests(MockTool(log_executive=True), mock_options)
		152	tool = MockTool(log_executive=True)
		153	# FIXME: We shouldn't use a real port-object here, but there is too much to mock at the moment.
		154	tool._deprecated_port = DeprecatedPort()
		155	step = steps.RunTests(tool, mock_options)
		156	expected_logs = """MOCK run_and_throw_if_fail: ['Tools/Scripts/run-javascriptcore-tests', '--no-fail-fast', '--json-output=jsc_test_results.json'], cwd=/mock-checkout
		157	"""
		158	OutputCapture().assert_outputs(self, step.run, [{}], expected_logs=expected_logs)
		159
		160	def test_runtests_jsc_debug(self):
		161	mock_options = self._step_options()
		162	mock_options.non_interactive = False
		163	mock_options.build_style = "debug"
		164	mock_options.group = "jsc"
		165	tool = MockTool(log_executive=True)
		166	# FIXME: We shouldn't use a real port-object here, but there is too much to mock at the moment.
		167	tool._deprecated_port = DeprecatedPort()
		168	step = steps.RunTests(tool, mock_options)
		169	expected_logs = """MOCK run_and_throw_if_fail: ['Tools/Scripts/run-javascriptcore-tests', '--no-fail-fast', '--json-output=jsc_test_results.json', '--debug'], cwd=/mock-checkout
		170	"""
		171	OutputCapture().assert_outputs(self, step.run, [{}], expected_logs=expected_logs)
		172
		173	def test_build_jsc_debug(self):
		174	mock_options = self._step_options()
		175	mock_options.non_interactive = False
		176	mock_options.build_style = "debug"
		177	mock_options.build = True
		178	mock_options.architecture = True
		179	mock_options.group = "jsc"
		180	tool = MockTool(log_executive=True)
		181	# FIXME: We shouldn't use a real port-object here, but there is too much to mock at the moment.
		182	tool._deprecated_port = DeprecatedPort()
		183	step = steps.Build(tool, mock_options)
		184	expected_logs = """Building WebKit
		185	MOCK run_and_throw_if_fail: ['Tools/Scripts/build-jsc', '--debug', 'ARCHS=True'], cwd=/mock-checkout, env={'LC_ALL': 'C', 'TERM': 'none', 'MOCK_ENVIRON_COPY': '1'}
		186	"""
		187	OutputCapture().assert_outputs(self, step.run, [{}], expected_logs=expected_logs)
		188
		189	def test_build_jsc(self):
		190	mock_options = self._step_options()
		191	mock_options.non_interactive = False
		192	mock_options.build_style = "release"
		193	mock_options.build = True
		194	mock_options.architecture = True
		195	mock_options.group = "jsc"
		196	tool = MockTool(log_executive=True)
		197	# FIXME: We shouldn't use a real port-object here, but there is too much to mock at the moment.
		198	tool._deprecated_port = DeprecatedPort()
		199	step = steps.Build(tool, mock_options)
		200	expected_logs = """Building WebKit
		201	MOCK run_and_throw_if_fail: ['Tools/Scripts/build-jsc', '--release', 'ARCHS=True'], cwd=/mock-checkout, env={'LC_ALL': 'C', 'TERM': 'none', 'MOCK_ENVIRON_COPY': '1'}
		202	"""
		203	OutputCapture().assert_outputs(self, step.run, [{}], expected_logs=expected_logs)
		204
		205	def test_patch_relevant(self):
		206	self.maxDiff = None
		207	mock_options = self._step_options()
		208	tool = MockTool(log_executive=True)
209	tool.scm()._mockChangedFiles = ["JSTests/MockFile1", "ChangeLog"]
210	# FIXME: We shouldn't use a real port-object here, but there is too much to mock at the moment.
211	tool._deprecated_port = DeprecatedPort()
212	step = steps.CheckPatchRelevance(tool, mock_options)
213	expected_logs = """Checking relevance of patch
214	This patch has relevant changes.
215	"""
216	OutputCapture().assert_outputs(self, step.run, [{}], expected_logs=expected_logs)
217
218	def test_patch_relevant_jsc(self):
219	self.maxDiff = None
220	mock_options = self._step_options()
221	mock_options.group = "jsc"
222	tool = MockTool(log_executive=True)
223	tool.scm()._mockChangedFiles = ["JSTests/MockFile1", "ChangeLog"]
224	# FIXME: We shouldn't use a real port-object here, but there is too much to mock at the moment.
225	tool._deprecated_port = DeprecatedPort()
226	step = steps.CheckPatchRelevance(tool, mock_options)
227	expected_logs = """Checking relevance of patch
228	This patch has relevant changes.
229	"""
230	OutputCapture().assert_outputs(self, step.run, [{}], expected_logs=expected_logs)
231
232	def test_patch_not_relevant_jsc(self):
233	self.maxDiff = None
234	mock_options = self._step_options()
235	mock_options.group = "jsc"
236	tool = MockTool(log_executive=True)
237	tool.scm()._mockChangedFiles = ["Tools/ChangeLog", "Tools/Scripts/webkitpy/tool/steps/steps_unittest.py"]
238	# FIXME: We shouldn't use a real port-object here, but there is too much to mock at the moment.
239	tool._deprecated_port = DeprecatedPort()
240	step = steps.CheckPatchRelevance(tool, mock_options)
241	expected_logs = """Checking relevance of patch
242	This patch does not have relevant changes.
243	"""
244	with self.assertRaises(ScriptError):
245	OutputCapture().assert_outputs(self, step.run, [{}], expected_logs=expected_logs)