Attachment #301437 for bug #162458

View Details Raw Unified Return to Bug 162458

| Differences between



2017-02-09  Srinivasan Vijayaraghavan  <svijayaraghavan@apple.com>

        EWS should run JavaScriptCore tests
        https://bugs.webkit.org/show_bug.cgi?id=162458

        Reviewed by NOBODY (OOPS!).

        * QueueStatusServer/config/queues.py: Add jsc-ews queue
        * QueueStatusServer/model/queuestatus.py:
        (QueueStatus.did_skip): Returns whether patch was skipped, based on status.
        * QueueStatusServer/handlers/statusbubble.py:
        (StatusBubble._should_show_bubble_for): Add logic to hide jsc-ews bubble if the patch does not touch jsc.
        * Scripts/webkitpy/common/checkout/scm/scm_mock.py:
        (MockSCM.__init__): Add _mockChangedFiles variable.
        (MockSCM.changed_files): Change to use _mockChangedFiles variables.
        * Scripts/webkitpy/common/config/ews.json: Add config info for JSC EWS.
        * Scripts/webkitpy/common/config/ports.py: Add support for JSC EWS in Mac Port.
        (DeprecatedPort.build_jsc_command): Added - command to build JSC only (quicker than building all of WebKit).
        (DeprecatedPort.run_javascriptcore_tests_command): Allow JSC EWS to only run JSC tests.
        (MacPort.run_webkit_tests_command): Check for JSC.
        * Scripts/webkitpy/common/config/ports_mock.py:
        (MockPort.run_javascriptcore_tests_command): Add build_style argument.
        * Scripts/webkitpy/common/config/ports_unittest.py:
        (DeprecatedPortTest.test_mac_port): Add unit tests for build-jsc shell command.
        * Scripts/webkitpy/common/net/jsctestresults.py: Added.
        (JSCTestResults): Added.
        (JSCTestResults.intersection): Return a JSCTestResults object with failures common to both input objects.
        (JSCTestResults.results_from_string): Use json library to parse results.
        (JSCTestResults.__init__): Initialize members.
        (JSCTestResults.equals): This enables unit testing.
        (JSCTestResults.is_subset): Checks if one set of failures is a subset of another.
        (JSCTestResults.all_passed): Getter.
        (JSCTestResults.failing_test_results): Getter. Some of these exist for interopability reasons.
        (JSCTestResults.test_results): Getter.
        (JSCTestResults.failing_tests): Getter.
        (JSCTestResults.did_exceed_test_failure_limit): Getter.
        * Scripts/webkitpy/common/net/jsctestresults_unittest.py: Added.
        (JSCTestResultsTest): Class to test JSCTestResults.
        (JSCTestResultsTest.test_results_from_string): Tests parsing.
        (JSCTestResultsTest.test_intersection_api_tests): Unit test for intersection() class method.
        (JSCTestResultsTest.test_intersection_stress_tests): Unit test for intersection() class method.
        (JSCTestResultsTest.test_intersection_general_case): Unit test for intersection() class method.
        * Scripts/webkitpy/tool/bot/earlywarningsystemtask.py:
        (EarlyWarningSystemTask.run): Add check_patch_relevance step.
        * Scripts/webkitpy/tool/bot/jscews_unittest.py: Added.
        (MockPatchAnalysisTask): Mocked-out version of PatchAnalysisTask that doesn't run commands.
        (MockPatchAnalysisTask.__init__): Sets attributes.
        (MockPatchAnalysisTask._test): Override to not run command.
        (MockPatchAnalysisTask._build_and_test_without_patch): Override to not run command.
        (MockPatchAnalysisTask.validate): Assume mocked patch is valid for purposes of testing retry logic.
        (MockPatchAnalysisTask.test_run_count): Specific to the mocked version, to test retry logic.
        (MockJSCEarlyWarningSystem): Mocked-out version of AbstarctEarlyWarningSystem so we can provide test results.
        (MockJSCEarlyWarningSystem.__init__): Sets attributes, also sets group to jsc.
        (MockJSCEarlyWarningSystem.test_results): Returns test results provided by us, instead of using a JSON reader.
        (JSCEarlyWarningSystemTest): Class to test retry logic in below situations.
        (JSCEarlyWarningSystemTest._create_task): Helper function to abstract out common code.
        (JSCEarlyWarningSystemTest._results_indicate_all_passed): False if input is None or has failures, else True.
        (JSCEarlyWarningSystemTest.test_success_case): Clean patch on clean tree.
        (JSCEarlyWarningSystemTest.test_test_failure): Failed patch on clean tree.
        (JSCEarlyWarningSystemTest.test_fix): Patch that fixes all tree redness.
        (JSCEarlyWarningSystemTest.test_ineffective_patch): Patch that has same failures as tree.
        (JSCEarlyWarningSystemTest.test_partially_effective_patch): Patch fixes some failures but adds no new failures.
        (JSCEarlyWarningSystemTest.test_different_test_failures_in_patch_and_tree): Patch has some failures not in tree.
        (JSCEarlyWarningSystemTest.test_first_results_could_not_be_read): Patch results not readable.
        (JSCEarlyWarningSystemTest.test_second_results_could_not_be_read): Patch results not readable on second run.
        (JSCEarlyWarningSystemTest.test_clean_results_could_not_be_read): Results from clean tree not readable.
        (JSCEarlyWarningSystemTest.test_flaky_results_on_clean_tree_pass): Patch has one flake and no failures.
        (JSCEarlyWarningSystemTest.test_flaky_results_on_clean_tree_pass_v2): Patch has one flake and no failures.
        (JSCEarlyWarningSystemTest.test_flaky_results_on_clean_tree_failure): Patch has flakes and failed tests.
        (JSCEarlyWarningSystemTest.test_flaky_results_on_red_tree_pass): Patch has same failures as tree, plus a flake.
        * Scripts/webkitpy/tool/bot/jsctestresultsreader.py: Added.
        (JSCTestResultsReader): Reads results file.
        (JSCTestResultsReader.__init__): Sets attributes.
        (JSCTestResultsReader._read_file_contents): Reads file.
        (JSCTestResultsReader._create_jsc_test_results): Creates a JSCTestResults object.
        (JSCTestResultsReader.results): Invokes _create_jsc_test_results to return a JSCTestResults object.
        * Scripts/webkitpy/tool/bot/patchanalysistask.py:
        (PatchIsNotApplicable): Exception for when patch doesn't have relevant changes.
        (PatchAnalysisTask._check_patch_relevance): Added.
        (PatchAnalysisTask._build): Check for JSC.
        (PatchAnalysisTask._build_without_patch): Check for JSC.
        (PatchAnalysisTask._test): Check for JSC.
        (PatchAnalysisTask._build_and_test_without_patch): Check for JSC.
        (PatchAnalysisTask._retry_jsc_tests): Retry logic for JSC EWS.
        (PatchAnalysisTask._retry_layout_tests): Retry logic for layout tests EWS.
        (PatchAnalysisTask._test_patch): Add retry logic for JSC.
        * Scripts/webkitpy/tool/commands/download.py:
        (CheckPatchRelevance): Add check-patch-relevance command.
        * Scripts/webkitpy/tool/commands/earlywarningsystem.py:
        (AbstractEarlyWarningSystem._create_task): Abstract out to enable mocking.
        (AbstractEarlyWarningSystem.begin_work_queue): Use JSCTestResultsReader not LayoutTestResultsReader in JSC EWS.
        (AbstractEarlyWarningSystem.review_patch): Handle PatchIsApplicable
        (AbstractEarlyWarningSystem.test_results): _layout_test_results_reader -> _test_results_reader.
        (AbstractEarlyWarningSystem.archive_last_test_results): _layout_test_results_reader -> _test_results_reader.
        (AbstractEarlyWarningSystem.group): This attribute determines the type of EWS (eg. JSC).
        (AbstractEarlyWarningSystem.load_ews_classes): Add _group, and make classes of type cls to enable mocking.
        * Scripts/webkitpy/tool/commands/earlywarningsystem_unittest.py:
        (TestEWS): Sample layout test EWS class used for unit testing.
        (TestJSCEWS): Sample JSC EWS class used for unit testing.
        (AbstractEarlyWarningSystemTest.test_failing_tests_message.TestEWS): Add _group.
        (AbstractEarlyWarningSystemTest.test_failing_jsc_tests_message.TestEWS): Added test for jsc failures message.
        (AbstractEarlyWarningSystemTest): Add _group variable.
        (EarlyWarningSystemTest._default_expected_logs): Add check-patch-relevance step, inconclusive logs, and group.
        (EarlyWarningSystemTest._test_ews): Add logs_are_conclusive option to pass through to default_expected_logs().
        (EarlyWarningSystemTest.test_inconclusive_test_results): Test not removing patch from queue if not conclusive.
        (MockAbstractEarlyWarningSystem): Added.
        (MockAbstractEarlyWarningSystem.test_results): Simulates inconclusive results.
        (MockAbstractEarlyWarningSystem._create_task): Use MockEarlyWarningSystemTask (not EarlyWarningSystemTask).
        (MockEarlyWarningSystemTask): Added.
        (MockEarlyWarningSystemTask._test_patch): Simulates running tests but not getting a conclusive result.
        * Scripts/webkitpy/tool/steps/__init__.py: Add CheckPatchRelevance import.
        * Scripts/webkitpy/tool/steps/build.py:
        (Build.options): Check for JSC.
        (Build.build): Check for JSC.
        (Build.run): Check for JSC.
        * Scripts/webkitpy/tool/steps/checkpatchrelevance.py: Added.
        (CheckPatchRelevance): Added.
        (CheckPatchRelevance._does_contain_change_in_paths): Abstract function to perform regex matching.
        (CheckPatchRelevance.run): Check if changed files in patch belong in certain folders.
        * Scripts/webkitpy/tool/steps/options.py:
        (Options): Add --group command line option.
        * Scripts/webkitpy/tool/steps/runtests.py:
        (RunTests.options): Add group.
        (RunTests.run): Check for JSC.
        (RunTests._run_webkit_tests): Check for JSC.
        (RunTests): Add _group attribute.
        (RunTests._run_javascriptcore_tests): New.
        * Scripts/webkitpy/tool/steps/steps_unittest.py: Unit tests.

2017-02-10  Joseph Pecoraro  <pecoraro@apple.com>

        [Resource Timing] Enable Resource Timing by default in Tests



# Copyright (C) 2013 Google Inc. All rights reserved.
# Copyright (C) 2017 Apple Inc. All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are

# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

# These must be in sync with webkit-patch's AbstractQueue.
skip_status = "Skip"
pass_status = "Pass"
fail_status = "Fail"
error_status = "Error"

Lines 1-4 a/Tools/QueueStatusServer/config/queues.py_sec1
- a/Tools/QueueStatusServer/config/queues.py -1 / +2 lines
1	# Copyright (C) 2014 Apple Inc. All rights reserved.	1	# Copyright (C) 2014, 2017 Apple Inc. All rights reserved.
2	# Copyright (C) 2013 Google Inc. All rights reserved.	2	# Copyright (C) 2013 Google Inc. All rights reserved.
3	#	3	#
4	# Redistribution and use in source and binary forms, with or without	4	# Redistribution and use in source and binary forms, with or without
Lines 36-41 all_queue_names = [ a/Tools/QueueStatusServer/config/queues.py_sec2
36	"gtk-wk2-ews",	36	"gtk-wk2-ews",
37	"ios-ews",	37	"ios-ews",
38	"ios-sim-ews",	38	"ios-sim-ews",
		39	"jsc-ews",
39	"mac-ews",	40	"mac-ews",
40	"mac-wk2-ews",	41	"mac-wk2-ews",
41	"mac-debug-ews",	42	"mac-debug-ews",

Lines 1-4 a/Tools/QueueStatusServer/handlers/statusbubble.py_sec1
- a/Tools/QueueStatusServer/handlers/statusbubble.py -3 / +9 lines
1	# Copyright (C) 2009 Google Inc. All rights reserved.	1	# Copyright (C) 2009 Google Inc. All rights reserved.
		2	# Copyright (C) 2017 Apple Inc. All rights reserved.
2	#	3	#
3	# Redistribution and use in source and binary forms, with or without	4	# Redistribution and use in source and binary forms, with or without
4	# modification, are permitted provided that the following conditions are	5	# modification, are permitted provided that the following conditions are
Lines 182-192 class StatusBubble(webapp.RequestHandler): a/Tools/QueueStatusServer/handlers/statusbubble.py_sec2
182	return bubble	183	return bubble
183		184
184	def _should_show_bubble_for(self, attachment, queue):	185	def _should_show_bubble_for(self, attachment, queue):
185	# Any pending queue is shown.	186	# Any pending queue is shown.
186	if attachment.position_in_queue(queue):	187	if attachment.position_in_queue(queue):
187	return True	188	return True
188	# EWS queues are also shown when complete.	189
189	return bool(queue.is_ews() and attachment.status_for_queue(queue))	190	# EWS queues that weren't skipped are also shown when complete.
		191	if not queue.is_ews():
		192	return False
		193
		194	status = attachment.status_for_queue(queue)
		195	return bool(status and not status.did_skip())
190		196
191	def _build_bubbles_for_attachment(self, attachment):	197	def _build_bubbles_for_attachment(self, attachment):
192	show_submit_to_ews = True	198	show_submit_to_ews = True

Lines 1-4 a/Tools/QueueStatusServer/model/queuestatus.py_sec1
- a/Tools/QueueStatusServer/model/queuestatus.py +4 lines
1	# Copyright (C) 2013 Google Inc. All rights reserved.	1	# Copyright (C) 2013 Google Inc. All rights reserved.
		2	# Copyright (C) 2017 Apple Inc. All rights reserved.
2	#	3	#
3	# Redistribution and use in source and binary forms, with or without	4	# Redistribution and use in source and binary forms, with or without
4	# modification, are permitted provided that the following conditions are	5	# modification, are permitted provided that the following conditions are
Lines 40-42 class QueueStatus(db.Model, QueuePropertyMixin): a/Tools/QueueStatusServer/model/queuestatus.py_sec2
40	message = db.StringProperty(multiline=True)	41	message = db.StringProperty(multiline=True)
41	date = db.DateTimeProperty(auto_now_add=True)	42	date = db.DateTimeProperty(auto_now_add=True)
42	results_file = db.BlobProperty()	43	results_file = db.BlobProperty()
		44
		45	def did_skip(self):
		46	return self.message.startswith(messages.skip_status)



# Copyright (C) 2011 Google Inc. All rights reserved.
# Copyright (C) 2017 Apple Inc. All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are

        self.added_paths = set()
        self._filesystem = filesystem or MockFileSystem()
        self._executive = executive or MockExecutive()
        self._mockChangedFiles = ["MockFile1"]

    def add(self, destination_path):
        self.add_list([destination_path])

        return self._filesystem.join(self.checkout_root, *comps)

    def changed_files(self, git_commit=None):
        return self._mockChangedFiles

    def changed_files_for_revision(self, revision):
        return ["MockFile1"]

Lines 50-54 a/Tools/Scripts/webkitpy/common/config/ews.json_sec1
- a/Tools/Scripts/webkitpy/common/config/ews.json +6 lines
50	"port": "mac",	50	"port": "mac",
51	"name": "mac-32bit-ews",	51	"name": "mac-32bit-ews",
52	"architecture": "i386"	52	"architecture": "i386"
		53	},
		54	"JSC EWS": {
		55	"port": "mac",
		56	"name": "jsc-ews",
		57	"group": "jsc",
		58	"runTests": true
53	}	59	}
54	}	60	}



# Copyright (C) 2009, Google Inc. All rights reserved.
# Copyright (C) 2013 Nokia Corporation and/or its subsidiary(-ies).
# Copyright (C) 2015, 2017 Apple Inc. All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are

            command.append("--release")
        return command

    def build_jsc_command(self, build_style=None):
        command = self.script_shell_command("build-jsc")
        if build_style == "debug":
            command.append("--debug")
        if build_style == "release":
            command.append("--release")
        return command

    def run_javascriptcore_tests_command(self, build_style=None):
        command = self.script_shell_command("run-javascriptcore-tests")
        command.append("--no-fail-fast")
        command.append("--json-output=jsc_test_results.json")
        if (build_style == "debug"):
            command.append("--debug")
        return command

    def run_webkit_tests_command(self, build_style=None):
        command = self.script_shell_command("run-webkit-tests")

class MacPort(DeprecatedPort):
    port_flag_name = "mac"

    def run_webkit_tests_command(self, build_style=None, test_suite=None):
        if test_suite == "jsc":
            return super(MacPort, self).run_javascriptcore_tests_command(build_style)

        command = super(MacPort, self).run_webkit_tests_command(build_style)
        command.append("--dump-render-tree")
        return command



# Copyright (C) 2011 Google Inc. All rights reserved.
# Copyright (C) 2017 Apple Inc. All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are

    def run_perl_unittests_command(self):
        return ['mock-test-webkitperl']

    def run_javascriptcore_tests_command(self, build_style=None):
        return ['mock-run-javacriptcore-tests']

    def run_webkit_tests_command(self, build_style=None):

Lines 1-4 a/Tools/Scripts/webkitpy/common/config/ports_unittest.py_sec1
- a/Tools/Scripts/webkitpy/common/config/ports_unittest.py +5 lines
1	# Copyright (c) 2009, Google Inc. All rights reserved.	1	# Copyright (c) 2009, Google Inc. All rights reserved.
		2	# Copyright (C) 2017 Apple Inc. All rights reserved.
2	#	3	#
3	# Redistribution and use in source and binary forms, with or without	4	# Redistribution and use in source and binary forms, with or without
4	# modification, are permitted provided that the following conditions are	5	# modification, are permitted provided that the following conditions are
Lines 39-44 class DeprecatedPortTest(unittest.TestCase): a/Tools/Scripts/webkitpy/common/config/ports_unittest.py_sec2
39	self.assertEqual(MacPort().build_webkit_command(build_style="debug"), DeprecatedPort().script_shell_command("build-webkit") + ["--debug"])	40	self.assertEqual(MacPort().build_webkit_command(build_style="debug"), DeprecatedPort().script_shell_command("build-webkit") + ["--debug"])
40	self.assertEqual(MacPort().build_webkit_command(build_style="release"), DeprecatedPort().script_shell_command("build-webkit") + ["--release"])	41	self.assertEqual(MacPort().build_webkit_command(build_style="release"), DeprecatedPort().script_shell_command("build-webkit") + ["--release"])
41		42
		43	self.assertEqual(MacPort().build_jsc_command(), DeprecatedPort().script_shell_command("build-jsc"))
		44	self.assertEqual(MacPort().build_jsc_command(build_style="release"), DeprecatedPort().script_shell_command("build-jsc") + ["--release"])
		45	self.assertEqual(MacPort().build_jsc_command(build_style="debug"), DeprecatedPort().script_shell_command("build-jsc") + ["--debug"])
		46
42	def test_gtk_wk2_port(self):	47	def test_gtk_wk2_port(self):
43	self.assertEqual(GtkWK2Port().flag(), "--port=gtk-wk2")	48	self.assertEqual(GtkWK2Port().flag(), "--port=gtk-wk2")
44	self.assertEqual(GtkWK2Port().run_webkit_tests_command(), DeprecatedPort().script_shell_command("run-webkit-tests") + ["--gtk"])	49	self.assertEqual(GtkWK2Port().run_webkit_tests_command(), DeprecatedPort().script_shell_command("run-webkit-tests") + ["--gtk"])



# Copyright (C) 2017 Apple Inc. All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions
# are met:
# 1.  Redistributions of source code must retain the above copyright
#     notice, this list of conditions and the following disclaimer.
# 2.  Redistributions in binary form must reproduce the above copyright
#     notice, this list of conditions and the following disclaimer in the
#     documentation and/or other materials provided with the distribution.
#
# THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS'' AND
# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
# DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS BE LIABLE FOR
# ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
# SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

import json
import logging

_log = logging.getLogger(__name__)


class JSCTestResults(object):
    def __init__(self, all_api_tests_passed, stress_test_failures):
        self.all_api_tests_passed = all_api_tests_passed
        self.stress_test_failures = stress_test_failures

        self.results = stress_test_failures[:]
        if not self.all_api_tests_passed:
            self.results.append("apiTests")

    @classmethod
    def intersection(cls, first, second):
        intersection_api_tests_passed = first.all_api_tests_passed or second.all_api_tests_passed
        intersection_stress_test_failures = list(set(first.stress_test_failures) & set(second.stress_test_failures))
        return cls(intersection_api_tests_passed, intersection_stress_test_failures)

    @classmethod
    def results_from_string(cls, string):
        if not string:
            return None

        try:
            parsed_results = json.loads(string)
        except ValueError:
            _log.error("Invalid JSON results")
            return None

        if 'allApiTestsPassed' not in parsed_results or 'stressTestFailures' not in parsed_results:
            return None

        return cls(parsed_results['allApiTestsPassed'], parsed_results['stressTestFailures'])

    def equals(self, other):
        return (self.all_api_tests_passed == other.all_api_tests_passed and
               set(self.stress_test_failures) == set(other.stress_test_failures))

    def is_subset(self, other):
        return set(self.results) <= set(other.results)

    def all_passed(self):
        return self.all_api_tests_passed and not self.stress_test_failures

    def failing_test_results(self):
        return self.results

    def test_results(self):
        return self.results

    def failing_tests(self):
        return self.results

    # No defined failure limit for JSC.
    def did_exceed_test_failure_limit(self):
        return False



# Copyright (C) 2017 Apple Inc. All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions
# are met:
# 1.  Redistributions of source code must retain the above copyright
#     notice, this list of conditions and the following disclaimer.
# 2.  Redistributions in binary form must reproduce the above copyright
#     notice, this list of conditions and the following disclaimer in the
#     documentation and/or other materials provided with the distribution.
#
# THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS'' AND
# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
# DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS BE LIABLE FOR
# ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
# SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

import unittest

from webkitpy.common.net.jsctestresults import JSCTestResults


class JSCTestResultsTest(unittest.TestCase):
    def test_results_from_string(self):
        none_item = None
        empty_json = ""
        invalid_json = '{"allApiTestsPassed":'
        incomplete_json_v1 = '{"allApiTestsPassed": true}'
        incomplete_json_v2 = '{"stressTestFailures":[]}'
        self.assertEqual(None, JSCTestResults.results_from_string(none_item))
        self.assertEqual(None, JSCTestResults.results_from_string(empty_json))
        self.assertEqual(None, JSCTestResults.results_from_string(invalid_json))
        self.assertEqual(None, JSCTestResults.results_from_string(incomplete_json_v1))
        self.assertEqual(None, JSCTestResults.results_from_string(incomplete_json_v2))

        no_failures_string = '{"allApiTestsPassed": true, "stressTestFailures":[]}'
        no_failures_results = JSCTestResults(True, [])
        self.assertTrue(no_failures_results.equals(JSCTestResults.results_from_string(no_failures_string)))

        api_test_failures_string = '{"allApiTestsPassed": false, "stressTestFailures":[]}'
        api_test_failures_results = JSCTestResults(False, [])
        self.assertTrue(api_test_failures_results.equals(JSCTestResults.results_from_string(api_test_failures_string)))

        many_failures_string = '{"allApiTestsPassed": false, "stressTestFailures":["es6.yaml/es6/typed_arrays_Int16Array.js.default", "es6.yaml/es6/typed_arrays_Int8Array.js.default"]}'
        many_failures_results = JSCTestResults(False, ["es6.yaml/es6/typed_arrays_Int16Array.js.default", "es6.yaml/es6/typed_arrays_Int8Array.js.default"])
        self.assertTrue(many_failures_results.equals(JSCTestResults.results_from_string(many_failures_string)))

        self.assertFalse(no_failures_results == api_test_failures_results)
        self.assertFalse(api_test_failures_results == many_failures_results)

    def test_intersection_api_tests(self):
        results1 = JSCTestResults(False, [])
        results2 = JSCTestResults(True, [])

        expected_intersection = JSCTestResults(True, [])
        self.assertTrue(expected_intersection.equals(JSCTestResults.intersection(results1, results2)))

    def test_intersection_stress_tests(self):
        results1 = JSCTestResults(True, ['failure1', 'failure2'])
        results2 = JSCTestResults(True, ['failure1', 'failure3'])

        expected_intersection = JSCTestResults(True, ['failure1'])
        self.assertTrue(expected_intersection.equals(JSCTestResults.intersection(results1, results2)))

    def test_intersection_general_case(self):
        results1 = JSCTestResults(True, ['failure1', 'failure2'])
        results2 = JSCTestResults(False, ['failure1'])

        expected_intersection = JSCTestResults(True, ['failure1'])
        self.assertTrue(expected_intersection.equals(JSCTestResults.intersection(results1, results2)))

Lines 1-4 a/Tools/Scripts/webkitpy/tool/bot/earlywarningsystemtask.py_sec1
- a/Tools/Scripts/webkitpy/tool/bot/earlywarningsystemtask.py -1 / +4 lines
1	# Copyright (c) 2011 Google Inc. All rights reserved.	1	# Copyright (c) 2011 Google Inc. All rights reserved.
		2	# Copyright (C) 2017 Apple Inc. All rights reserved.
2	#	3	#
3	# Redistribution and use in source and binary forms, with or without	4	# Redistribution and use in source and binary forms, with or without
4	# modification, are permitted provided that the following conditions are	5	# modification, are permitted provided that the following conditions are
Lines 26-32 a/Tools/Scripts/webkitpy/tool/bot/earlywarningsystemtask.py_sec2
26	# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE	27	# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
27	# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.	28	# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
28		29
29	from webkitpy.tool.bot.patchanalysistask import PatchAnalysisTask, PatchAnalysisTaskDelegate, UnableToApplyPatch, PatchIsNotValid	30	from webkitpy.tool.bot.patchanalysistask import PatchAnalysisTask, PatchAnalysisTaskDelegate, UnableToApplyPatch, PatchIsNotValid, PatchIsNotApplicable
30		31
31		32
32	class EarlyWarningSystemTaskDelegate(PatchAnalysisTaskDelegate):	33	class EarlyWarningSystemTaskDelegate(PatchAnalysisTaskDelegate):
Lines 58-63 class EarlyWarningSystemTask(PatchAnalysisTask): a/Tools/Scripts/webkitpy/tool/bot/earlywarningsystemtask.py_sec3
58	return False	59	return False
59	if not self._apply():	60	if not self._apply():
60	raise UnableToApplyPatch(self._patch)	61	raise UnableToApplyPatch(self._patch)
		62	if not self._check_patch_relevance():
		63	raise PatchIsNotApplicable(self._patch)
61	if not self._build():	64	if not self._build():
62	if not self._build_without_patch():	65	if not self._build_without_patch():
63	return False	66	return False



# Copyright (C) 2017 Apple Inc. All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions
# are met:
# 1.  Redistributions of source code must retain the above copyright
#     notice, this list of conditions and the following disclaimer.
# 2.  Redistributions in binary form must reproduce the above copyright
#     notice, this list of conditions and the following disclaimer in the
#     documentation and/or other materials provided with the distribution.
#
# THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS'' AND
# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
# DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS BE LIABLE FOR
# ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
# SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

import logging
import unittest

from webkitpy.common.net.jsctestresults import JSCTestResults
from webkitpy.common.system.executive import ScriptError
from webkitpy.tool.bot.patchanalysistask import *
from webkitpy.tool.commands.earlywarningsystem import AbstractEarlyWarningSystem
from webkitpy.tool.mocktool import MockTool

_log = logging.getLogger(__name__)


class MockPatchAnalysisTask(PatchAnalysisTask):
    def __init__(self, delegate, patch, patches_passed_all_tests):
        self._delegate = delegate
        self._patch = patch
        self._script_error = None
        self._results_archive_from_patch_test_run = None
        self._results_from_patch_test_run = None
        self.error = None
        self._patches_passed_all_tests = patches_passed_all_tests
        self._test_run_count = 0
        self.failure_status_id = 0

    def _test(self):
        self._test_run_count += 1

        if self._patches_passed_all_tests.pop() == True:
            return True
        self._script_error = ScriptError("Regression test")
        return False

    def _build_and_test_without_patch(self):
        self._test_run_count += 1
        return True

    def validate(self):
        return True

    def test_run_count(self):
        return self._test_run_count


# This is the delegate to be used with MockPatchAnalysisTask, above.
class MockJSCEarlyWarningSystem(AbstractEarlyWarningSystem):
    def __init__(self, first_test_results, second_test_results, clean_test_results):
        AbstractEarlyWarningSystem.__init__(self)
        self._group = "jsc"
        self._results_in_order = [clean_test_results, second_test_results, first_test_results]

    def test_results(self):
        return self._results_in_order.pop()


class JSCEarlyWarningSystemTest(unittest.TestCase):
    def _results_indicate_all_passed(self, results):
        if results == None:
            return False
        return results.all_passed()

    def _create_task(self, first_test_results, second_test_results, clean_test_results):
        queue = MockJSCEarlyWarningSystem(first_test_results, second_test_results, clean_test_results)
        tool = MockTool(log_executive=True)
        patch = tool.bugs.fetch_attachment(10000)
        patches_passed_all_tests = map(self._results_indicate_all_passed, [second_test_results, first_test_results])
        return MockPatchAnalysisTask(queue, patch, patches_passed_all_tests)

    def test_success_case(self):
        first_test_results = JSCTestResults(True, [])
        second_test_results = JSCTestResults(True, [])
        clean_test_results = JSCTestResults(True, [])
        task = self._create_task(first_test_results, second_test_results, clean_test_results)

        return_value = task._test_patch()
        self.assertEqual(task.test_run_count(), 1)
        self.assertTrue(return_value)

    def test_test_failure(self):
        first_test_results = JSCTestResults(True, ['Fail.js'])
        second_test_results = JSCTestResults(True, ['Fail.js'])
        clean_test_results = JSCTestResults(True, [])
        task = self._create_task(first_test_results, second_test_results, clean_test_results)

        with self.assertRaises(ScriptError):
            return_value = task._test_patch()
        self.assertEqual(task.test_run_count(), 3)

    def test_fix(self):
        first_test_results = JSCTestResults(True, [])
        second_test_results = JSCTestResults(True, [])
        clean_test_results = JSCTestResults(True, ['Fail.js'])
        task = self._create_task(first_test_results, second_test_results, clean_test_results)

        return_value = task._test_patch()
        self.assertEqual(task.test_run_count(), 1)
        self.assertTrue(return_value)

    def test_ineffective_patch(self):
        first_test_results = JSCTestResults(False, ['failure1.js', 'failure2.js'])
        second_test_results = JSCTestResults(False, ['failure1.js', 'failure2.js'])
        clean_test_results = JSCTestResults(False, ['failure1.js', 'failure2.js'])
        task = self._create_task(first_test_results, second_test_results, clean_test_results)

        return_value = task._test_patch()
        self.assertEqual(task.test_run_count(), 3)
        self.assertTrue(return_value)

    def test_partially_effective_patch(self):
        first_test_results = JSCTestResults(True, ['failure2.js'])
        second_test_results = JSCTestResults(True, ['failure2.js'])
        clean_test_results = JSCTestResults(False, ['failure1.js', 'failure2.js'])
        task = self._create_task(first_test_results, second_test_results, clean_test_results)

        return_value = task._test_patch()
        self.assertEqual(task.test_run_count(), 3)
        self.assertTrue(return_value)

    def test_different_test_failures_in_patch_and_tree(self):
        first_test_results = JSCTestResults(False, [])
        second_test_results = JSCTestResults(False, [])
        clean_test_results = JSCTestResults(True, ['failure1.js', 'failure2.js'])
        task = self._create_task(first_test_results, second_test_results, clean_test_results)

        with self.assertRaises(ScriptError):
            return_value = task._test_patch()
        self.assertEqual(task.test_run_count(), 3)

    def test_first_results_could_not_be_read(self):
        first_test_results = None
        second_test_results = JSCTestResults(True, [])
        clean_test_results = JSCTestResults(True, [])
        task = self._create_task(first_test_results, second_test_results, clean_test_results)

        return_value = task._test_patch()
        self.assertEqual(task.test_run_count(), 1)
        self.assertFalse(return_value)

    def test_second_results_could_not_be_read(self):
        first_test_results = JSCTestResults(False, ['failure1.js', 'failure2.js'])
        second_test_results = None
        clean_test_results = JSCTestResults(True, [])
        task = self._create_task(first_test_results, second_test_results, clean_test_results)

        return_value = task._test_patch()
        self.assertEqual(task.test_run_count(), 2)
        self.assertFalse(return_value)

    def test_clean_results_could_not_be_read(self):
        first_test_results = JSCTestResults(True, ['failure2.js'])
        second_test_results = JSCTestResults(True, ['failure2.js'])
        clean_test_results = None
        task = self._create_task(first_test_results, second_test_results, clean_test_results)

        return_value = task._test_patch()
        self.assertEqual(task.test_run_count(), 3)
        self.assertFalse(return_value)

    def test_flaky_results_on_clean_tree_pass(self):
        first_test_results = JSCTestResults(True, ['failure2.js'])
        second_test_results = JSCTestResults(True, [])
        clean_test_results = JSCTestResults(True, [])
        task = self._create_task(first_test_results, second_test_results, clean_test_results)

        return_value = task._test_patch()
        self.assertTrue(return_value)
        self.assertEqual(task.test_run_count(), 2)

    def test_flaky_results_on_clean_tree_pass_v2(self):
        first_test_results = JSCTestResults(True, [])
        second_test_results = JSCTestResults(True, ['failure2.js'])
        clean_test_results = JSCTestResults(True, [])
        task = self._create_task(first_test_results, second_test_results, clean_test_results)

        return_value = task._test_patch()
        self.assertTrue(return_value)
        self.assertEqual(task.test_run_count(), 1)

    def test_flaky_results_on_clean_tree_failure(self):
        first_test_results = JSCTestResults(False, ['failure1.js', 'failure2.js'])
        second_test_results = JSCTestResults(True, ['failure2.js'])
        clean_test_results = JSCTestResults(True, [])
        task = self._create_task(first_test_results, second_test_results, clean_test_results)

        with self.assertRaises(ScriptError):
            task._test_patch()
        self.assertEqual(task.test_run_count(), 3)

    def test_flaky_results_on_red_tree_pass(self):
        first_test_results = JSCTestResults(True, ['failure1.js'])
        second_test_results = JSCTestResults(True, ['failure1.js', 'failure2.js'])
        clean_test_results = JSCTestResults(True, ['failure1.js'])
        task = self._create_task(first_test_results, second_test_results, clean_test_results)

        return_value = task._test_patch()
        self.assertTrue(return_value)
        self.assertEqual(task.test_run_count(), 3)



# Copyright (C) 2017 Apple Inc. All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions
# are met:
# 1.  Redistributions of source code must retain the above copyright
#     notice, this list of conditions and the following disclaimer.
# 2.  Redistributions in binary form must reproduce the above copyright
#     notice, this list of conditions and the following disclaimer in the
#     documentation and/or other materials provided with the distribution.
#
# THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS'' AND
# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
# DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS BE LIABLE FOR
# ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
# SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

import logging

from webkitpy.common.net.jsctestresults import JSCTestResults

_log = logging.getLogger(__name__)


class JSCTestResultsReader(object):
    def __init__(self, host, results_directory):
        self._host = host
        self._results_directory = results_directory

    def _read_file_contents(self, path):
        try:
            return self._host.filesystem.read_text_file(path)
        except (IOError, KeyError):  # File does not exist or can't be read.
            return None

    def _create_jsc_test_results(self):
        results_path = self._host.filesystem.join(self._results_directory, "jsc_test_results.json")
        contents = self._read_file_contents(results_path)
        return JSCTestResults.results_from_string(contents)

    def results(self):
        return self._create_jsc_test_results()

Lines 1-4 a/Tools/Scripts/webkitpy/tool/bot/patchanalysistask.py_sec1
- a/Tools/Scripts/webkitpy/tool/bot/patchanalysistask.py -21 / +88 lines
1	# Copyright (c) 2010 Google Inc. All rights reserved.	1	# Copyright (c) 2010 Google Inc. All rights reserved.
		2	# Copyright (C) 2017 Apple Inc. All rights reserved.
2	#	3	#
3	# Redistribution and use in source and binary forms, with or without	4	# Redistribution and use in source and binary forms, with or without
4	# modification, are permitted provided that the following conditions are	5	# modification, are permitted provided that the following conditions are
Lines 28-33 a/Tools/Scripts/webkitpy/tool/bot/patchanalysistask.py_sec2
28		29
29	from webkitpy.common.system.executive import ScriptError	30	from webkitpy.common.system.executive import ScriptError
30	from webkitpy.common.net.layouttestresults import LayoutTestResults	31	from webkitpy.common.net.layouttestresults import LayoutTestResults
		32	from webkitpy.common.net.jsctestresults import JSCTestResults
31		33
32		34
33	class UnableToApplyPatch(Exception):	35	class UnableToApplyPatch(Exception):
Lines 43-48 class PatchIsNotValid(Exception): a/Tools/Scripts/webkitpy/tool/bot/patchanalysistask.py_sec3
43	self.failure_message = failure_message	45	self.failure_message = failure_message
44		46
45		47
		48	class PatchIsNotApplicable(Exception):
		49	def __init__(self, patch):
		50	Exception.__init__(self)
		51	self.patch = patch
		52
46	class PatchAnalysisTaskDelegate(object):	53	class PatchAnalysisTaskDelegate(object):
47	def parent_command(self):	54	def parent_command(self):
48	raise NotImplementedError("subclasses must implement")	55	raise NotImplementedError("subclasses must implement")
Lines 122-149 class PatchAnalysisTask(object): a/Tools/Scripts/webkitpy/tool/bot/patchanalysistask.py_sec4
122	"Applied patch",	129	"Applied patch",
123	"Patch does not apply")	130	"Patch does not apply")
124		131
		132	def _check_patch_relevance(self):
		133	args = [
		134	"check-patch-relevance",
		135	]
		136
		137	if hasattr(self._delegate, 'group'):
		138	args.append("--group=%s" % self._delegate.group())
		139
		140	return self._run_command(args, "Checked relevance of patch", "Patch was not relevant")
		141
125	def _build(self):	142	def _build(self):
126	return self._run_command([	143	args = [
127	"build",	144	"build",
128	"--no-clean",	145	"--no-clean",
129	"--no-update",	146	"--no-update",
130	"--build-style=%s" % self._delegate.build_style(),	147	"--build-style=%s" % self._delegate.build_style(),
131	],	148	]
132	"Built patch",	149
133	"Patch does not build")	150	if hasattr(self._delegate, 'group'):
		151	args.append("--group=%s" % self._delegate.group())
		152
		153	return self._run_command(args, "Built patch", "Patch does not build")
134		154
135	def _build_without_patch(self):	155	def _build_without_patch(self):
136	return self._run_command([	156	args = [
137	"build",	157	"build",
138	"--force-clean",	158	"--force-clean",
139	"--no-update",	159	"--no-update",
140	"--build-style=%s" % self._delegate.build_style(),	160	"--build-style=%s" % self._delegate.build_style(),
141	],	161	]
142	"Able to build without patch",	162
143	"Unable to build without patch")	163	if hasattr(self._delegate, 'group'):
		164	args.append("--group=%s" % self._delegate.group())
		165
		166	return self._run_command(args, "Able to build without patch", "Unable to build without patch")
144		167
145	def _test(self):	168	def _test(self):
146	return self._run_command([	169	args = [
147	"build-and-test",	170	"build-and-test",
148	"--no-clean",	171	"--no-clean",
149	"--no-update",	172	"--no-update",
Lines 151-162 class PatchAnalysisTask(object): a/Tools/Scripts/webkitpy/tool/bot/patchanalysistask.py_sec5
151	"--test",	174	"--test",
152	"--non-interactive",	175	"--non-interactive",
153	"--build-style=%s" % self._delegate.build_style(),	176	"--build-style=%s" % self._delegate.build_style(),
154	],	177	]
155	"Passed tests",	178
156	"Patch does not pass tests")	179	if hasattr(self._delegate, 'group'):
		180	args.append("--group=%s" % self._delegate.group())
		181
		182	return self._run_command(args, "Passed tests", "Patch does not pass tests")
157		183
158	def _build_and_test_without_patch(self):	184	def _build_and_test_without_patch(self):
159	return self._run_command([	185	args = [
160	"build-and-test",	186	"build-and-test",
161	"--force-clean",	187	"--force-clean",
162	"--no-update",	188	"--no-update",
Lines 164-172 class PatchAnalysisTask(object): a/Tools/Scripts/webkitpy/tool/bot/patchanalysistask.py_sec6
164	"--test",	190	"--test",
165	"--non-interactive",	191	"--non-interactive",
166	"--build-style=%s" % self._delegate.build_style(),	192	"--build-style=%s" % self._delegate.build_style(),
167	],	193	]
168	"Able to pass tests without patch",	194
169	"Unable to pass tests without patch (tree is red?)")	195	if hasattr(self._delegate, 'group'):
		196	args.append("--group=%s" % self._delegate.group())
		197
		198	return self._run_command(args, "Able to pass tests without patch", "Unable to pass tests without patch (tree is red?)")
170		199
171	def _land(self):	200	def _land(self):
172	# Unclear if this should pass --quiet or not. If --parent-command always does the reporting, then it should.	201	# Unclear if this should pass --quiet or not. If --parent-command always does the reporting, then it should.
Lines 206-215 class PatchAnalysisTask(object): a/Tools/Scripts/webkitpy/tool/bot/patchanalysistask.py_sec7
206	# also present without the patch, so we don't need to defer.	235	# also present without the patch, so we don't need to defer.
207	return False	236	return False
208		237
209	def _test_patch(self):	238	# FIXME: Abstract out common parts of the retry logic.
		239	def _retry_jsc_tests(self):
		240	first_results = self._delegate.test_results()
		241	first_script_error = self._script_error
		242	first_failure_status_id = self.failure_status_id
		243	if first_results is None:
		244	return False
		245
210	if self._test():	246	if self._test():
211	return True	247	return True
		248	second_results = self._delegate.test_results()
		249	second_script_error = self._script_error
		250	if second_results is None:
		251	return False
		252
		253	consistent_results = JSCTestResults.intersection(first_results, second_results)
		254
		255	self._build_and_test_without_patch()
		256	clean_tree_results = self._delegate.test_results()
		257	if clean_tree_results is None:
		258	return False
		259
		260	if consistent_results.is_subset(clean_tree_results):
		261	return True
		262
		263	self.failure_status_id = first_failure_status_id
		264	return self.report_failure(None, consistent_results, first_script_error)
212		265
		266	def _retry_layout_tests(self):
213	# Note: archive_last_test_results deletes the results directory, making these calls order-sensitve.	267	# Note: archive_last_test_results deletes the results directory, making these calls order-sensitve.
214	# We could remove this dependency by building the test_results from the archive.	268	# We could remove this dependency by building the test_results from the archive.
215	first_results = self._delegate.test_results()	269	first_results = self._delegate.test_results()
Lines 240-250 class PatchAnalysisTask(object): a/Tools/Scripts/webkitpy/tool/bot/patchanalysistask.py_sec8
240	return self.report_failure(first_results_archive, first_results, first_script_error)	294	return self.report_failure(first_results_archive, first_results, first_script_error)
241		295
242	if second_results.did_exceed_test_failure_limit():	296	if second_results.did_exceed_test_failure_limit():
243	self._should_defer_patch_or_throw(first_results.failing_test_results(), first_results_archive, first_script_error, first_failure_status_id)	297	self._should_defer_patch_or_throw(first_results.failing_test_results(), first_results_archive,
		298	first_script_error, first_failure_status_id)
244	return False	299	return False
245		300
246	if first_results.did_exceed_test_failure_limit():	301	if first_results.did_exceed_test_failure_limit():
247	self._should_defer_patch_or_throw(second_results.failing_test_results(), second_results_archive, second_script_error, second_failure_status_id)	302	self._should_defer_patch_or_throw(second_results.failing_test_results(), second_results_archive,
		303	second_script_error, second_failure_status_id)
248	return False	304	return False
249		305
250	if self._results_failed_different_tests(first_results, second_results):	306	if self._results_failed_different_tests(first_results, second_results):
Lines 259-272 class PatchAnalysisTask(object): a/Tools/Scripts/webkitpy/tool/bot/patchanalysistask.py_sec9
259		315
260	tests_that_consistently_failed = first_failing_results_set.intersection(second_failing_results_set)	316	tests_that_consistently_failed = first_failing_results_set.intersection(second_failing_results_set)
261	if tests_that_consistently_failed:	317	if tests_that_consistently_failed:
262	if self._should_defer_patch_or_throw(tests_that_consistently_failed, first_results_archive, first_script_error, first_failure_status_id):	318	if self._should_defer_patch_or_throw(tests_that_consistently_failed, first_results_archive,
		319	first_script_error, first_failure_status_id):
263	return False # Defer patch	320	return False # Defer patch
264		321
265	# At this point we know that at least one test flaked, but no consistent failures	322	# At this point we know that at least one test flaked, but no consistent failures
266	# were introduced. This is a bit of a grey-zone.	323	# were introduced. This is a bit of a grey-zone.
267	return False # Defer patch	324	return False # Defer patch
268		325
269	if self._should_defer_patch_or_throw(first_results.failing_test_results(), first_results_archive, first_script_error, first_failure_status_id):	326	if self._should_defer_patch_or_throw(first_results.failing_test_results(), first_results_archive,
		327	first_script_error, first_failure_status_id):
270	return False # Defer patch	328	return False # Defer patch
271		329
272	# At this point, we know that the first and second runs had the exact same failures,	330	# At this point, we know that the first and second runs had the exact same failures,
Lines 274-279 class PatchAnalysisTask(object): a/Tools/Scripts/webkitpy/tool/bot/patchanalysistask.py_sec10
274	# that the patch is good.	332	# that the patch is good.
275	return True	333	return True
276		334
		335	def _test_patch(self):
		336	if self._test():
		337	return True
		338
		339	if hasattr(self._delegate, 'group') and self._delegate.group() == "jsc":
		340	return self._retry_jsc_tests()
		341	else:
		342	return self._retry_layout_tests()
		343
277	def results_archive_from_patch_test_run(self, patch):	344	def results_archive_from_patch_test_run(self, patch):
278	assert(self._patch.id() == patch.id()) # PatchAnalysisTask is not currently re-useable.	345	assert(self._patch.id() == patch.id()) # PatchAnalysisTask is not currently re-useable.
279	return self._results_archive_from_patch_test_run	346	return self._results_archive_from_patch_test_run

Lines 1-5 a/Tools/Scripts/webkitpy/tool/commands/download.py_sec1
- a/Tools/Scripts/webkitpy/tool/commands/download.py -1 / +9 lines
1	# Copyright (c) 2009, 2011 Google Inc. All rights reserved.	1	# Copyright (c) 2009, 2011 Google Inc. All rights reserved.
2	# Copyright (c) 2009 Apple Inc. All rights reserved.	2	# Copyright (c) 2009, 2017 Apple Inc. All rights reserved.
3	#	3	#
4	# Redistribution and use in source and binary forms, with or without	4	# Redistribution and use in source and binary forms, with or without
5	# modification, are permitted provided that the following conditions are	5	# modification, are permitted provided that the following conditions are
Lines 87-92 class BuildAndTest(AbstractSequencedCommand): a/Tools/Scripts/webkitpy/tool/commands/download.py_sec2
87	]	87	]
88		88
89		89
		90	class CheckPatchRelevance(AbstractSequencedCommand):
		91	name = "check-patch-relevance"
		92	help_text = "Check if this patch needs to be tested"
		93	steps = [
		94	steps.CheckPatchRelevance,
		95	]
		96
		97
90	class Land(AbstractSequencedCommand):	98	class Land(AbstractSequencedCommand):
91	name = "land"	99	name = "land"
92	help_text = "Land the current working directory diff and updates the associated bug if any"	100	help_text = "Land the current working directory diff and updates the associated bug if any"

Lines 89-94 class DownloadCommandsTest(CommandsTest): a/Tools/Scripts/webkitpy/tool/commands/download_unittest.py_sec1
- a/Tools/Scripts/webkitpy/tool/commands/download_unittest.py +1 lines
89	options.update = True	89	options.update = True
90	options.architecture = 'MOCK ARCH'	90	options.architecture = 'MOCK ARCH'
91	options.iterate_on_new_tests = 0	91	options.iterate_on_new_tests = 0
		92	options.group = None
92	return options	93	return options
93		94
94	def test_build(self):	95	def test_build(self):



# Copyright (c) 2009 Google Inc. All rights reserved.
# Copyright (c) 2017 Apple Inc. All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are


import json
import logging
import os
from optparse import make_option

from webkitpy.common.config.committers import CommitterList

from webkitpy.common.system.executive import ScriptError
from webkitpy.tool.bot.earlywarningsystemtask import EarlyWarningSystemTask, EarlyWarningSystemTaskDelegate
from webkitpy.tool.bot.layouttestresultsreader import LayoutTestResultsReader
from webkitpy.tool.bot.jsctestresultsreader import JSCTestResultsReader
from webkitpy.tool.bot.patchanalysistask import UnableToApplyPatch, PatchIsNotValid, PatchIsNotApplicable
from webkitpy.tool.bot.queueengine import QueueEngine
from webkitpy.tool.commands.queues import AbstractReviewQueue



    def begin_work_queue(self):
        AbstractReviewQueue.begin_work_queue(self)

        if self.group() == "jsc":
            jsc_results_directory = os.path.dirname(os.path.dirname(os.path.dirname(self._port.results_directory())))
            self._test_results_reader = JSCTestResultsReader(self._tool, jsc_results_directory)
        else:
            self._test_results_reader = LayoutTestResultsReader(self._tool, self._port.results_directory(), self._log_directory())

    def _failing_tests_message(self, task, patch):
        results = task.results_from_patch_test_run(patch)

            tool.bugs.add_cc_to_bug(patch.bug_id(), self.watchers)
        tool.bugs.set_flag_on_attachment(patch.id(), "commit-queue", "-", message)

    # This exists for mocking
    def _create_task(self, patch):
        return EarlyWarningSystemTask(self, patch, self._options.run_tests)

    def review_patch(self, patch):
        task = self._create_task(patch)
        try:
            succeeded = task.run()
            if not succeeded:

        except UnableToApplyPatch, e:
            self._did_error(patch, "%s unable to apply patch." % self.name)
            return False
        except PatchIsNotApplicable, e:
            self._did_skip(patch, "Patch not applicable to %s." % self.name)
            return False
        except ScriptError, e:
            self._post_reject_message_on_bug(self._tool, patch, task.failure_status_id, self._failing_tests_message(task, patch))
            results_archive = task.results_archive_from_patch_test_run(patch)

        return self._update_status(message, patch=patch, results_file=failure_log)

    def test_results(self):
        return self._test_results_reader.results()

    def archive_last_test_results(self, patch):
        return self._test_results_reader.archive(patch)

    def build_style(self):
        return self._build_style

    def group(self):
        return self._group

    def refetch_patch(self, patch):
        return self._tool.bugs.fetch_attachment(patch.id())



        classes = []
        for name, config in ewses.iteritems():
            classes.append(type(name.encode('utf-8').translate(None, ' -'), (cls,), {
                'name': config.get('name', config['port'] + '-ews'),
                'port_name': config['port'],
                'architecture': config.get('architecture', None),
                '_build_style': config.get('style', "release"),
                'watchers': config.get('watchers', []),
                'run_tests': config.get('runTests', cls.run_tests),
                '_group': config.get('group', None),
            }))
        return classes



# Copyright (C) 2009 Google Inc. All rights reserved.
# Copyright (C) 2017 Apple Inc. All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are

from webkitpy.thirdparty.mock import Mock
from webkitpy.common.host import Host
from webkitpy.common.host_mock import MockHost
from webkitpy.common.net.jsctestresults import JSCTestResults
from webkitpy.common.net.layouttestresults import LayoutTestResults
from webkitpy.common.system.outputcapture import OutputCapture
from webkitpy.layout_tests.models import test_results

from webkitpy.tool.mocktool import MockTool, MockOptions


# Needed to define port_name, used in AbstractEarlyWarningSystem.__init__
class TestEWS(AbstractEarlyWarningSystem):
    port_name = "win"  # Needs to be a port which port/factory understands.
    _build_style = None
    _group = None
            _build_style = None


class TestJSCEWS(AbstractEarlyWarningSystem):
    port_name = "mac"  # Needs to be a port which port/factory understands.
    _build_style = None
    _group = "jsc"


class AbstractEarlyWarningSystemTest(QueuesTest):
    def _test_message(self, ews, results, message):
        ews.bind_to_tool(MockTool())
        ews.host = MockHost()
        ews._options = MockOptions(port=None, confirm=False)
        OutputCapture().assert_outputs(self, ews.begin_work_queue, expected_logs=self._default_begin_work_queue_logs(ews.name))
        task = Mock()
        task.results_from_patch_test_run = results
                                                                          test_results.TestResult("bar.html", failures=[test_failures.FailureTextMismatch()])],
                                                                          did_exceed_test_failure_limit=False)
        patch = ews._tool.bugs.fetch_attachment(10000)
        self.assertMultiLineEqual(ews._failing_tests_message(task, patch), message)

    def test_failing_tests_message(self):
        ews = TestEWS()
        results = lambda a: LayoutTestResults([test_results.TestResult("foo.html", failures=[test_failures.FailureTextMismatch()]),
                                                test_results.TestResult("bar.html", failures=[test_failures.FailureTextMismatch()])],
                                                did_exceed_test_failure_limit=False)
        message = "New failing tests:\nfoo.html\nbar.html"
        self._test_message(ews, results, message)

    def test_failing_jsc_tests_message(self):
        ews = TestJSCEWS()
        results = lambda a: JSCTestResults(False, ["es6.yaml/es6/typed_arrays_Int8Array.js.default", "es6.yaml/es6/typed_arrays_Uint8Array.js.default"])
        message = "New failing tests:\nes6.yaml/es6/typed_arrays_Int8Array.js.default\nes6.yaml/es6/typed_arrays_Uint8Array.js.default\napiTests"
        self._test_message(ews, results, message)


class MockEarlyWarningSystemTaskForInconclusiveResults(EarlyWarningSystemTask):
    def _test_patch(self):
        self._test()
        return False


class MockAbstractEarlyWarningSystemForInconclusiveResults(AbstractEarlyWarningSystem):
    def test_results(self):
        return None

    def _create_task(self, patch):
        task = MockEarlyWarningSystemTaskForInconclusiveResults(self, patch, self._options.run_tests)
        return task


class EarlyWarningSystemTest(QueuesTest):
    def _default_expected_logs(self, ews, conclusive):
        string_replacements = {
            "name": ews.name,
            "port": ews.port_name,
            "architecture": " --architecture=%s" % ews.architecture if ews.architecture else "",
            "build_style": ews.build_style(),
            "group": ews.group(),
        }
        if ews.run_tests:
            run_tests_line = "Running: webkit-patch --status-host=example.com build-and-test --no-clean --no-update --test --non-interactive --build-style=%(build_style)s --group=%(group)s --port=%(port)s%(architecture)s\n" % string_replacements
        else:
            run_tests_line = ""
        string_replacements['run_tests_line'] = run_tests_line

        if conclusive:
            result_lines = "MOCK: update_status: %(name)s Pass\nMOCK: release_work_item: %(name)s 10000\n" % string_replacements
        else:
            result_lines = "MOCK: release_lock: %(name)s 10000\n" % string_replacements
        string_replacements['result_lines'] = result_lines

        expected_logs = {
            "begin_work_queue": self._default_begin_work_queue_logs(ews.name),
            "process_work_item": """MOCK: update_status: %(name)s Started processing patch
Running: webkit-patch --status-host=example.com clean --port=%(port)s%(architecture)s
Running: webkit-patch --status-host=example.com update --port=%(port)s%(architecture)s
Running: webkit-patch --status-host=example.com apply-attachment --no-update --non-interactive 10000 --port=%(port)s%(architecture)s
Running: webkit-patch --status-host=example.com check-patch-relevance --group=%(group)s --port=%(port)s%(architecture)s
Running: webkit-patch --status-host=example.com build --no-clean --no-update --build-style=%(build_style)s --group=%(group)s --port=%(port)s%(architecture)s
%(run_tests_line)s%(result_lines)s""" % string_replacements,
""" % string_replacements,
            "handle_unexpected_error": "Mock error message\n",
            "handle_script_error": "ScriptError error message\n\nMOCK output\n",
        }
        return expected_logs

    def _test_ews(self, ews, results_are_conclusive=True):
        ews.bind_to_tool(MockTool())
        ews.host = MockHost()
        options = Mock()
        options.port = None
        options.run_tests = ews.run_tests
        self.assert_queue_outputs(ews, expected_logs=self._default_expected_logs(ews, results_are_conclusive), options=options)

    def test_ewses(self):
        classes = AbstractEarlyWarningSystem.load_ews_classes()

        self.maxDiff = None
        for ews_class in classes:
            self._test_ews(ews_class())

    # This must use the Mock EWS classes defined above
    def test_inconclusive_test_results(self):
        classes = MockAbstractEarlyWarningSystemForInconclusiveResults.load_ews_classes()
        self.assertTrue(classes)
        self.maxDiff = None
        test_classes = filter(lambda x: x.run_tests, classes)
        for ews_class in test_classes:
            self._test_ews(ews_class(), False)

Lines 1-5 a/Tools/Scripts/webkitpy/tool/commands/queues.py_sec1
- a/Tools/Scripts/webkitpy/tool/commands/queues.py -1 / +7 lines
1	# Copyright (c) 2009 Google Inc. All rights reserved.	1	# Copyright (c) 2009 Google Inc. All rights reserved.
2	# Copyright (c) 2009 Apple Inc. All rights reserved.	2	# Copyright (c) 2009, 2017 Apple Inc. All rights reserved.
3	#	3	#
4	# Redistribution and use in source and binary forms, with or without	4	# Redistribution and use in source and binary forms, with or without
5	# modification, are permitted provided that the following conditions are	5	# modification, are permitted provided that the following conditions are
Lines 62-67 class AbstractQueue(Command, QueueEngineDelegate): a/Tools/Scripts/webkitpy/tool/commands/queues.py_sec2
62	watchers = [	62	watchers = [
63	]	63	]
64		64
		65	_skip_status = "Skip"
65	_pass_status = "Pass"	66	_pass_status = "Pass"
66	_fail_status = "Fail"	67	_fail_status = "Fail"
67	_error_status = "Error"	68	_error_status = "Error"
Lines 244-249 class AbstractPatchQueue(AbstractQueue): a/Tools/Scripts/webkitpy/tool/commands/queues.py_sec3
244	self._update_status(message, patch)	245	self._update_status(message, patch)
245	self._release_work_item(patch)	246	self._release_work_item(patch)
246		247
		248	def _did_skip(self, patch, reason):
		249	message = "%s: %s" % (self._skip_status, reason)
		250	self._update_status(message, patch)
		251	self._release_work_item(patch)
		252
247	def _unlock_patch(self, patch):	253	def _unlock_patch(self, patch):
248	self._tool.status_server.release_lock(self.name, patch)	254	self._tool.status_server.release_lock(self.name, patch)
249		255



# Copyright (C) 2010 Google Inc. All rights reserved.
# Copyright (C) 2017 Apple Inc. All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are

from webkitpy.tool.steps.applywatchlist import ApplyWatchList
from webkitpy.tool.steps.attachtobug import AttachToBug
from webkitpy.tool.steps.build import Build
from webkitpy.tool.steps.checkpatchrelevance import CheckPatchRelevance
from webkitpy.tool.steps.checkstyle import CheckStyle
from webkitpy.tool.steps.cleanworkingdirectory import CleanWorkingDirectory
from webkitpy.tool.steps.closebug import CloseBug



# Copyright (C) 2010 Google Inc. All rights reserved.
# Copyright (C) 2017 Apple Inc. All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are

            Options.build,
            Options.quiet,
            Options.build_style,
            Options.group,
        ]

    def build(self, build_style, group):
        environment = self._tool.copy_current_environment()
        environment.disable_gcc_smartquotes()
        environment.disable_jhbuild_VT100_output()
        env = environment.to_dictionary()

        if group == "jsc":
            build_command = self._tool.deprecated_port().build_jsc_command(build_style=build_style)
        else:
            build_command = self._tool.deprecated_port().build_webkit_command(build_style=build_style)

        if self._options.architecture:
            build_command += ['ARCHS=%s' % self._options.architecture]

        self._tool.executive.run_and_throw_if_fail(build_command, self._options.quiet,
            cwd=self._tool.scm().checkout_root, env=env)

    def run(self, state):
        if not self._options.build:
            return
        _log.info("Building WebKit")

        group = self._options.group

        if self._options.build_style == "both":
            self.build("debug", group)
            self.build("release", group)
        else:
            self.build(self._options.build_style, group)



# Copyright (C) 2017 Apple Inc. All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions
# are met:
# 1.  Redistributions of source code must retain the above copyright
#     notice, this list of conditions and the following disclaimer.
# 2.  Redistributions in binary form must reproduce the above copyright
#     notice, this list of conditions and the following disclaimer in the
#     documentation and/or other materials provided with the distribution.
#
# THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS'' AND
# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
# DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS BE LIABLE FOR
# ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
# SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

import logging
import re

from webkitpy.common.system.executive import ScriptError
from webkitpy.tool.steps.abstractstep import AbstractStep
from webkitpy.tool.steps.options import Options

_log = logging.getLogger(__name__)


class CheckPatchRelevance(AbstractStep):
    @classmethod
    def options(cls):
        return AbstractStep.options() + [
            Options.group,
        ]

    jsc_paths = [
        "JSTests/",
        "Source/JavaScriptCore/",
        "Source/WTF/"
        "Source/bmalloc/",
    ]

    group_to_paths_mapping = {
        'jsc': jsc_paths,
    }

    def _changes_are_relevant(self, changed_files):
        # In the default case, all patches are relevant
        if self._options.group is None:
            return True

        patterns = self.group_to_paths_mapping[self._options.group]

        for changed_file in changed_files:
            for pattern in patterns:
                if re.search(pattern, changed_file, re.IGNORECASE):
                    return True

        return False

    def run(self, state):
        _log.info("Checking relevance of patch")

        change_list = self._tool.scm().changed_files()

        if self._changes_are_relevant(change_list):
            _log.info("This patch has relevant changes.")
            return True

        _log.info("This patch does not have relevant changes.")
        raise ScriptError(message="This patch does not have relevant changes.")



# Copyright (C) 2010 Google Inc. All rights reserved.
# Copyright (C) 2017 Apple Inc. All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are

    email = make_option("--email", action="store", type="string", dest="email", help="Email address to use in ChangeLogs.")
    force_clean = make_option("--force-clean", action="store_true", dest="force_clean", default=False, help="Clean working directory before applying patches (removes local changes and commits)")
    git_commit = make_option("-g", "--git-commit", action="store", dest="git_commit", help="Operate on a local commit. If a range, the commits are squashed into one. <ref>.... includes the working copy changes. UPSTREAM can be used for the upstream/tracking branch.")
    group = make_option("--group", action="store", dest="group", default=None, help="")
    local_commit = make_option("--local-commit", action="store_true", dest="local_commit", default=False, help="Make a local commit for each applied patch")
    non_interactive = make_option("--non-interactive", action="store_true", dest="non_interactive", default=False, help="Never prompt the user, fail as fast as possible.")
    obsolete_patches = make_option("--no-obsolete", action="store_false", dest="obsolete_patches", default=True, help="Do not obsolete old patches before posting this one.")

Lines 1-4 a/Tools/Scripts/webkitpy/tool/steps/runtests.py_sec1
- a/Tools/Scripts/webkitpy/tool/steps/runtests.py +10 lines
1	# Copyright (C) 2010 Google Inc. All rights reserved.	1	# Copyright (C) 2010 Google Inc. All rights reserved.
		2	# Copyright (C) 2017 Apple Inc. All rights reserved.
2	#	3	#
3	# Redistribution and use in source and binary forms, with or without	4	# Redistribution and use in source and binary forms, with or without
4	# modification, are permitted provided that the following conditions are	5	# modification, are permitted provided that the following conditions are
Lines 53-61 class RunTests(AbstractStep): a/Tools/Scripts/webkitpy/tool/steps/runtests.py_sec2
53	Options.iterate_on_new_tests,	54	Options.iterate_on_new_tests,
54	Options.non_interactive,	55	Options.non_interactive,
55	Options.quiet,	56	Options.quiet,
		57	Options.group,
56	]	58	]
57		59
58	def run(self, state):	60	def run(self, state):
		61	if self._options.group == "jsc":
		62	self._run_javascriptcore_tests()
		63	return
		64
59	if self._options.iterate_on_new_tests:	65	if self._options.iterate_on_new_tests:
60	_log.info("Running run-webkit-tests on new tests")	66	_log.info("Running run-webkit-tests on new tests")
61	self._run_webkit_tests(self._options.iterate_on_new_tests)	67	self._run_webkit_tests(self._options.iterate_on_new_tests)
Lines 143-145 class RunTests(AbstractStep): a/Tools/Scripts/webkitpy/tool/steps/runtests.py_sec3
143	args.append("--iterations=%d" % iterate_on_new_tests)	149	args.append("--iterations=%d" % iterate_on_new_tests)
144		150
145	self._tool.executive.run_and_throw_if_fail(args, cwd=self._tool.scm().checkout_root)	151	self._tool.executive.run_and_throw_if_fail(args, cwd=self._tool.scm().checkout_root)
		152
		153	def _run_javascriptcore_tests(self):
		154	args = self._tool.deprecated_port().run_javascriptcore_tests_command(self._options.build_style)
		155	self._tool.executive.run_and_throw_if_fail(args, cwd=self._tool.scm().checkout_root)

Lines 40-46 class RunTestsTest(unittest.TestCase): a/Tools/Scripts/webkitpy/tool/steps/runtests_unittest.py_sec1
- a/Tools/Scripts/webkitpy/tool/steps/runtests_unittest.py -1 / +1 lines
40	tool = MockTool(log_executive=True)	40	tool = MockTool(log_executive=True)
41	tool._deprecated_port.run_python_unittests_command = lambda: None	41	tool._deprecated_port.run_python_unittests_command = lambda: None
42	tool._deprecated_port.run_perl_unittests_command = lambda: None	42	tool._deprecated_port.run_perl_unittests_command = lambda: None
43	step = RunTests(tool, MockOptions(test=True, non_interactive=True, quiet=False, build_style="release", iterate_on_new_tests=0))	43	step = RunTests(tool, MockOptions(test=True, non_interactive=True, quiet=False, build_style="release", iterate_on_new_tests=0, group=None))
44		44
45	if sys.platform != "cygwin":	45	if sys.platform != "cygwin":
46	expected_logs = """Running bindings generation tests	46	expected_logs = """Running bindings generation tests



# Copyright (C) 2010 Google Inc. All rights reserved.
# Copyright (C) 2017 Apple Inc. All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are


import unittest

from webkitpy.common.system.executive import ScriptError
from webkitpy.common.system.outputcapture import OutputCapture
from webkitpy.common.config.ports import DeprecatedPort
from webkitpy.tool.mocktool import MockOptions, MockTool

class StepsTest(unittest.TestCase):
    def _step_options(self):
        options = MockOptions()
        options.group = None
        options.non_interactive = True
        options.port = 'MOCK port'
        options.quiet = True

Running Perl unit tests
MOCK run_and_throw_if_fail: ['Tools/Scripts/test-webkitperl'], cwd=/mock-checkout
Running JavaScriptCore tests
MOCK run_and_throw_if_fail: ['Tools/Scripts/run-javascriptcore-tests', '--no-fail-fast', '--json-output=jsc_test_results.json'], cwd=/mock-checkout
Running bindings generation tests
MOCK run_and_throw_if_fail: ['Tools/Scripts/run-bindings-tests'], cwd=/mock-checkout
Running run-webkit-tests

Running Perl unit tests
MOCK run_and_throw_if_fail: ['Tools/Scripts/test-webkitperl'], cwd=/mock-checkout
Running JavaScriptCore tests
MOCK run_and_throw_if_fail: ['Tools/Scripts/run-javascriptcore-tests', '--no-fail-fast', '--json-output=jsc_test_results.json'], cwd=/mock-checkout
Running bindings generation tests
MOCK run_and_throw_if_fail: ['Tools/Scripts/run-bindings-tests'], cwd=/mock-checkout
Running run-webkit-tests
MOCK run_and_throw_if_fail: ['Tools/Scripts/run-webkit-tests', '--debug', '--quiet'], cwd=/mock-checkout
"""
        OutputCapture().assert_outputs(self, step.run, [{}], expected_logs=expected_logs)

    def test_runtests_jsc(self):
        mock_options = self._step_options()
        mock_options.non_interactive = False
        mock_options.build_style = "release"
        mock_options.group = "jsc"
        step = steps.RunTests(MockTool(log_executive=True), mock_options)
        tool = MockTool(log_executive=True)
        # FIXME: We shouldn't use a real port-object here, but there is too much to mock at the moment.
        tool._deprecated_port = DeprecatedPort()
        step = steps.RunTests(tool, mock_options)
        expected_logs = """MOCK run_and_throw_if_fail: ['Tools/Scripts/run-javascriptcore-tests', '--no-fail-fast', '--json-output=jsc_test_results.json'], cwd=/mock-checkout
"""
        OutputCapture().assert_outputs(self, step.run, [{}], expected_logs=expected_logs)

    def test_runtests_jsc_debug(self):
        mock_options = self._step_options()
        mock_options.non_interactive = False
        mock_options.build_style = "debug"
        mock_options.group = "jsc"
        tool = MockTool(log_executive=True)
        # FIXME: We shouldn't use a real port-object here, but there is too much to mock at the moment.
        tool._deprecated_port = DeprecatedPort()
        step = steps.RunTests(tool, mock_options)
        expected_logs = """MOCK run_and_throw_if_fail: ['Tools/Scripts/run-javascriptcore-tests', '--no-fail-fast', '--json-output=jsc_test_results.json', '--debug'], cwd=/mock-checkout
"""
        OutputCapture().assert_outputs(self, step.run, [{}], expected_logs=expected_logs)

    def test_build_jsc_debug(self):
        mock_options = self._step_options()
        mock_options.non_interactive = False
        mock_options.build_style = "debug"
        mock_options.build = True
        mock_options.architecture = True
        mock_options.group = "jsc"
        tool = MockTool(log_executive=True)
        # FIXME: We shouldn't use a real port-object here, but there is too much to mock at the moment.
        tool._deprecated_port = DeprecatedPort()
        step = steps.Build(tool, mock_options)
        expected_logs = """Building WebKit
MOCK run_and_throw_if_fail: ['Tools/Scripts/build-jsc', '--debug', 'ARCHS=True'], cwd=/mock-checkout, env={'LC_ALL': 'C', 'TERM': 'none', 'MOCK_ENVIRON_COPY': '1'}
"""
        OutputCapture().assert_outputs(self, step.run, [{}], expected_logs=expected_logs)

    def test_build_jsc(self):
        mock_options = self._step_options()
        mock_options.non_interactive = False
        mock_options.build_style = "release"
        mock_options.build = True
        mock_options.architecture = True
        mock_options.group = "jsc"
        tool = MockTool(log_executive=True)
        # FIXME: We shouldn't use a real port-object here, but there is too much to mock at the moment.
        tool._deprecated_port = DeprecatedPort()
        step = steps.Build(tool, mock_options)
        expected_logs = """Building WebKit
MOCK run_and_throw_if_fail: ['Tools/Scripts/build-jsc', '--release', 'ARCHS=True'], cwd=/mock-checkout, env={'LC_ALL': 'C', 'TERM': 'none', 'MOCK_ENVIRON_COPY': '1'}
"""
        OutputCapture().assert_outputs(self, step.run, [{}], expected_logs=expected_logs)

    def test_patch_relevant(self):
        self.maxDiff = None
        mock_options = self._step_options()
        tool = MockTool(log_executive=True)
        tool.scm()._mockChangedFiles = ["JSTests/MockFile1", "ChangeLog"]
        # FIXME: We shouldn't use a real port-object here, but there is too much to mock at the moment.
        tool._deprecated_port = DeprecatedPort()
        step = steps.CheckPatchRelevance(tool, mock_options)
        expected_logs = """Checking relevance of patch
This patch has relevant changes.
"""
        OutputCapture().assert_outputs(self, step.run, [{}], expected_logs=expected_logs)

    def test_patch_relevant_jsc(self):
        self.maxDiff = None
        mock_options = self._step_options()
        mock_options.group = "jsc"
        tool = MockTool(log_executive=True)
        tool.scm()._mockChangedFiles = ["JSTests/MockFile1", "ChangeLog"]
        # FIXME: We shouldn't use a real port-object here, but there is too much to mock at the moment.
        tool._deprecated_port = DeprecatedPort()
        step = steps.CheckPatchRelevance(tool, mock_options)
        expected_logs = """Checking relevance of patch
This patch has relevant changes.
"""
        OutputCapture().assert_outputs(self, step.run, [{}], expected_logs=expected_logs)

    def test_patch_not_relevant_jsc(self):
        self.maxDiff = None
        mock_options = self._step_options()
        mock_options.group = "jsc"
        tool = MockTool(log_executive=True)
        tool.scm()._mockChangedFiles = ["Tools/ChangeLog", "Tools/Scripts/webkitpy/tool/steps/steps_unittest.py"]
        # FIXME: We shouldn't use a real port-object here, but there is too much to mock at the moment.
        tool._deprecated_port = DeprecatedPort()
        step = steps.CheckPatchRelevance(tool, mock_options)
        expected_logs = """Checking relevance of patch
This patch does not have relevant changes.
"""
        with self.assertRaises(ScriptError):
            OutputCapture().assert_outputs(self, step.run, [{}], expected_logs=expected_logs)

Return to Bug 162458

Lines 1-3 a/Tools/ChangeLog_sec1
- a/Tools/ChangeLog +129 lines
		1	2017-02-09 Srinivasan Vijayaraghavan <svijayaraghavan@apple.com>
		2
		3	EWS should run JavaScriptCore tests
		4	https://bugs.webkit.org/show_bug.cgi?id=162458
		5
		6	Reviewed by NOBODY (OOPS!).
		7
		8	* QueueStatusServer/config/queues.py: Add jsc-ews queue
		9	* QueueStatusServer/model/queuestatus.py:
		10	(QueueStatus.did_skip): Returns whether patch was skipped, based on status.
		11	* QueueStatusServer/handlers/statusbubble.py:
		12	(StatusBubble._should_show_bubble_for): Add logic to hide jsc-ews bubble if the patch does not touch jsc.
		13	* Scripts/webkitpy/common/checkout/scm/scm_mock.py:
		14	(MockSCM.__init__): Add _mockChangedFiles variable.
		15	(MockSCM.changed_files): Change to use _mockChangedFiles variables.
		16	* Scripts/webkitpy/common/config/ews.json: Add config info for JSC EWS.
		17	* Scripts/webkitpy/common/config/ports.py: Add support for JSC EWS in Mac Port.
		18	(DeprecatedPort.build_jsc_command): Added - command to build JSC only (quicker than building all of WebKit).
		19	(DeprecatedPort.run_javascriptcore_tests_command): Allow JSC EWS to only run JSC tests.
		20	(MacPort.run_webkit_tests_command): Check for JSC.
		21	* Scripts/webkitpy/common/config/ports_mock.py:
		22	(MockPort.run_javascriptcore_tests_command): Add build_style argument.
		23	* Scripts/webkitpy/common/config/ports_unittest.py:
		24	(DeprecatedPortTest.test_mac_port): Add unit tests for build-jsc shell command.
		25	* Scripts/webkitpy/common/net/jsctestresults.py: Added.
		26	(JSCTestResults): Added.
		27	(JSCTestResults.intersection): Return a JSCTestResults object with failures common to both input objects.
		28	(JSCTestResults.results_from_string): Use json library to parse results.
		29	(JSCTestResults.__init__): Initialize members.
		30	(JSCTestResults.equals): This enables unit testing.
		31	(JSCTestResults.is_subset): Checks if one set of failures is a subset of another.
		32	(JSCTestResults.all_passed): Getter.
		33	(JSCTestResults.failing_test_results): Getter. Some of these exist for interopability reasons.
		34	(JSCTestResults.test_results): Getter.
		35	(JSCTestResults.failing_tests): Getter.
		36	(JSCTestResults.did_exceed_test_failure_limit): Getter.
		37	* Scripts/webkitpy/common/net/jsctestresults_unittest.py: Added.
		38	(JSCTestResultsTest): Class to test JSCTestResults.
		39	(JSCTestResultsTest.test_results_from_string): Tests parsing.
		40	(JSCTestResultsTest.test_intersection_api_tests): Unit test for intersection() class method.
		41	(JSCTestResultsTest.test_intersection_stress_tests): Unit test for intersection() class method.
		42	(JSCTestResultsTest.test_intersection_general_case): Unit test for intersection() class method.
		43	* Scripts/webkitpy/tool/bot/earlywarningsystemtask.py:
		44	(EarlyWarningSystemTask.run): Add check_patch_relevance step.
		45	* Scripts/webkitpy/tool/bot/jscews_unittest.py: Added.
		46	(MockPatchAnalysisTask): Mocked-out version of PatchAnalysisTask that doesn't run commands.
		47	(MockPatchAnalysisTask.__init__): Sets attributes.
		48	(MockPatchAnalysisTask._test): Override to not run command.
		49	(MockPatchAnalysisTask._build_and_test_without_patch): Override to not run command.
		50	(MockPatchAnalysisTask.validate): Assume mocked patch is valid for purposes of testing retry logic.
		51	(MockPatchAnalysisTask.test_run_count): Specific to the mocked version, to test retry logic.
		52	(MockJSCEarlyWarningSystem): Mocked-out version of AbstarctEarlyWarningSystem so we can provide test results.
		53	(MockJSCEarlyWarningSystem.__init__): Sets attributes, also sets group to jsc.
		54	(MockJSCEarlyWarningSystem.test_results): Returns test results provided by us, instead of using a JSON reader.
		55	(JSCEarlyWarningSystemTest): Class to test retry logic in below situations.
		56	(JSCEarlyWarningSystemTest._create_task): Helper function to abstract out common code.
		57	(JSCEarlyWarningSystemTest._results_indicate_all_passed): False if input is None or has failures, else True.
		58	(JSCEarlyWarningSystemTest.test_success_case): Clean patch on clean tree.
		59	(JSCEarlyWarningSystemTest.test_test_failure): Failed patch on clean tree.
		60	(JSCEarlyWarningSystemTest.test_fix): Patch that fixes all tree redness.
		61	(JSCEarlyWarningSystemTest.test_ineffective_patch): Patch that has same failures as tree.
		62	(JSCEarlyWarningSystemTest.test_partially_effective_patch): Patch fixes some failures but adds no new failures.
		63	(JSCEarlyWarningSystemTest.test_different_test_failures_in_patch_and_tree): Patch has some failures not in tree.
		64	(JSCEarlyWarningSystemTest.test_first_results_could_not_be_read): Patch results not readable.
65	(JSCEarlyWarningSystemTest.test_second_results_could_not_be_read): Patch results not readable on second run.
66	(JSCEarlyWarningSystemTest.test_clean_results_could_not_be_read): Results from clean tree not readable.
67	(JSCEarlyWarningSystemTest.test_flaky_results_on_clean_tree_pass): Patch has one flake and no failures.
68	(JSCEarlyWarningSystemTest.test_flaky_results_on_clean_tree_pass_v2): Patch has one flake and no failures.
69	(JSCEarlyWarningSystemTest.test_flaky_results_on_clean_tree_failure): Patch has flakes and failed tests.
70	(JSCEarlyWarningSystemTest.test_flaky_results_on_red_tree_pass): Patch has same failures as tree, plus a flake.
71	* Scripts/webkitpy/tool/bot/jsctestresultsreader.py: Added.
72	(JSCTestResultsReader): Reads results file.
73	(JSCTestResultsReader.__init__): Sets attributes.
74	(JSCTestResultsReader._read_file_contents): Reads file.
75	(JSCTestResultsReader._create_jsc_test_results): Creates a JSCTestResults object.
76	(JSCTestResultsReader.results): Invokes _create_jsc_test_results to return a JSCTestResults object.
77	* Scripts/webkitpy/tool/bot/patchanalysistask.py:
78	(PatchIsNotApplicable): Exception for when patch doesn't have relevant changes.
79	(PatchAnalysisTask._check_patch_relevance): Added.
80	(PatchAnalysisTask._build): Check for JSC.
81	(PatchAnalysisTask._build_without_patch): Check for JSC.
82	(PatchAnalysisTask._test): Check for JSC.
83	(PatchAnalysisTask._build_and_test_without_patch): Check for JSC.
84	(PatchAnalysisTask._retry_jsc_tests): Retry logic for JSC EWS.
85	(PatchAnalysisTask._retry_layout_tests): Retry logic for layout tests EWS.
86	(PatchAnalysisTask._test_patch): Add retry logic for JSC.
87	* Scripts/webkitpy/tool/commands/download.py:
88	(CheckPatchRelevance): Add check-patch-relevance command.
89	* Scripts/webkitpy/tool/commands/earlywarningsystem.py:
90	(AbstractEarlyWarningSystem._create_task): Abstract out to enable mocking.
91	(AbstractEarlyWarningSystem.begin_work_queue): Use JSCTestResultsReader not LayoutTestResultsReader in JSC EWS.
92	(AbstractEarlyWarningSystem.review_patch): Handle PatchIsApplicable
93	(AbstractEarlyWarningSystem.test_results): _layout_test_results_reader -> _test_results_reader.
94	(AbstractEarlyWarningSystem.archive_last_test_results): _layout_test_results_reader -> _test_results_reader.
95	(AbstractEarlyWarningSystem.group): This attribute determines the type of EWS (eg. JSC).
96	(AbstractEarlyWarningSystem.load_ews_classes): Add _group, and make classes of type cls to enable mocking.
97	* Scripts/webkitpy/tool/commands/earlywarningsystem_unittest.py:
98	(TestEWS): Sample layout test EWS class used for unit testing.
99	(TestJSCEWS): Sample JSC EWS class used for unit testing.
100	(AbstractEarlyWarningSystemTest.test_failing_tests_message.TestEWS): Add _group.
101	(AbstractEarlyWarningSystemTest.test_failing_jsc_tests_message.TestEWS): Added test for jsc failures message.
102	(AbstractEarlyWarningSystemTest): Add _group variable.
103	(EarlyWarningSystemTest._default_expected_logs): Add check-patch-relevance step, inconclusive logs, and group.
104	(EarlyWarningSystemTest._test_ews): Add logs_are_conclusive option to pass through to default_expected_logs().
105	(EarlyWarningSystemTest.test_inconclusive_test_results): Test not removing patch from queue if not conclusive.
106	(MockAbstractEarlyWarningSystem): Added.
107	(MockAbstractEarlyWarningSystem.test_results): Simulates inconclusive results.
108	(MockAbstractEarlyWarningSystem._create_task): Use MockEarlyWarningSystemTask (not EarlyWarningSystemTask).
109	(MockEarlyWarningSystemTask): Added.
110	(MockEarlyWarningSystemTask._test_patch): Simulates running tests but not getting a conclusive result.
111	* Scripts/webkitpy/tool/steps/__init__.py: Add CheckPatchRelevance import.
112	* Scripts/webkitpy/tool/steps/build.py:
113	(Build.options): Check for JSC.
114	(Build.build): Check for JSC.
115	(Build.run): Check for JSC.
116	* Scripts/webkitpy/tool/steps/checkpatchrelevance.py: Added.
117	(CheckPatchRelevance): Added.
118	(CheckPatchRelevance._does_contain_change_in_paths): Abstract function to perform regex matching.
119	(CheckPatchRelevance.run): Check if changed files in patch belong in certain folders.
120	* Scripts/webkitpy/tool/steps/options.py:
121	(Options): Add --group command line option.
122	* Scripts/webkitpy/tool/steps/runtests.py:
123	(RunTests.options): Add group.
124	(RunTests.run): Check for JSC.
125	(RunTests._run_webkit_tests): Check for JSC.
126	(RunTests): Add _group attribute.
127	(RunTests._run_javascriptcore_tests): New.
128	* Scripts/webkitpy/tool/steps/steps_unittest.py: Unit tests.
129
1	2017-02-10 Joseph Pecoraro <pecoraro@apple.com>	130	2017-02-10 Joseph Pecoraro <pecoraro@apple.com>
2		131
3	[Resource Timing] Enable Resource Timing by default in Tests	132	[Resource Timing] Enable Resource Timing by default in Tests

Lines 1-4 a/Tools/QueueStatusServer/config/messages.py_sec1
- a/Tools/QueueStatusServer/config/messages.py +2 lines
1	# Copyright (C) 2013 Google Inc. All rights reserved.	1	# Copyright (C) 2013 Google Inc. All rights reserved.
		2	# Copyright (C) 2017 Apple Inc. All rights reserved.
2	#	3	#
3	# Redistribution and use in source and binary forms, with or without	4	# Redistribution and use in source and binary forms, with or without
4	# modification, are permitted provided that the following conditions are	5	# modification, are permitted provided that the following conditions are
Lines 27-32 a/Tools/QueueStatusServer/config/messages.py_sec2
27	# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.	28	# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
28		29
29	# These must be in sync with webkit-patch's AbstractQueue.	30	# These must be in sync with webkit-patch's AbstractQueue.
		31	skip_status = "Skip"
30	pass_status = "Pass"	32	pass_status = "Pass"
31	fail_status = "Fail"	33	fail_status = "Fail"
32	error_status = "Error"	34	error_status = "Error"

Lines 1-4 a/Tools/Scripts/webkitpy/common/checkout/scm/scm_mock.py_sec1
- a/Tools/Scripts/webkitpy/common/checkout/scm/scm_mock.py -1 / +3 lines
1	# Copyright (C) 2011 Google Inc. All rights reserved.	1	# Copyright (C) 2011 Google Inc. All rights reserved.
		2	# Copyright (C) 2017 Apple Inc. All rights reserved.
2	#	3	#
3	# Redistribution and use in source and binary forms, with or without	4	# Redistribution and use in source and binary forms, with or without
4	# modification, are permitted provided that the following conditions are	5	# modification, are permitted provided that the following conditions are
Lines 37-42 class MockSCM(object): a/Tools/Scripts/webkitpy/common/checkout/scm/scm_mock.py_sec2
37	self.added_paths = set()	38	self.added_paths = set()
38	self._filesystem = filesystem or MockFileSystem()	39	self._filesystem = filesystem or MockFileSystem()
39	self._executive = executive or MockExecutive()	40	self._executive = executive or MockExecutive()
		41	self._mockChangedFiles = ["MockFile1"]
40		42
41	def add(self, destination_path):	43	def add(self, destination_path):
42	self.add_list([destination_path])	44	self.add_list([destination_path])
Lines 71-77 class MockSCM(object): a/Tools/Scripts/webkitpy/common/checkout/scm/scm_mock.py_sec3
71	return self._filesystem.join(self.checkout_root, *comps)	73	return self._filesystem.join(self.checkout_root, *comps)
72		74
73	def changed_files(self, git_commit=None):	75	def changed_files(self, git_commit=None):
74	return ["MockFile1"]	76	return self._mockChangedFiles
75		77
76	def changed_files_for_revision(self, revision):	78	def changed_files_for_revision(self, revision):
77	return ["MockFile1"]	79	return ["MockFile1"]

Line 0 a/Tools/Scripts/webkitpy/common/net/jsctestresults.py_sec1
- a/Tools/Scripts/webkitpy/common/net/jsctestresults.py +81 lines
		1	# Copyright (C) 2017 Apple Inc. All rights reserved.
		2	#
		3	# Redistribution and use in source and binary forms, with or without
		4	# modification, are permitted provided that the following conditions
		5	# are met:
		6	# 1. Redistributions of source code must retain the above copyright
		7	# notice, this list of conditions and the following disclaimer.
		8	# 2. Redistributions in binary form must reproduce the above copyright
		9	# notice, this list of conditions and the following disclaimer in the
		10	# documentation and/or other materials provided with the distribution.
		11	#
		12	# THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS'' AND
		13	# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
		14	# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
		15	# DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS BE LIABLE FOR
		16	# ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
		17	# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
		18	# SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
		19	# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
		20	# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
		21	# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
		22
		23	import json
		24	import logging
		25
		26	_log = logging.getLogger(__name__)
		27
		28
		29	class JSCTestResults(object):
		30	def __init__(self, all_api_tests_passed, stress_test_failures):
		31	self.all_api_tests_passed = all_api_tests_passed
		32	self.stress_test_failures = stress_test_failures
		33
		34	self.results = stress_test_failures[:]
		35	if not self.all_api_tests_passed:
		36	self.results.append("apiTests")
		37
		38	@classmethod
		39	def intersection(cls, first, second):
		40	intersection_api_tests_passed = first.all_api_tests_passed or second.all_api_tests_passed
		41	intersection_stress_test_failures = list(set(first.stress_test_failures) & set(second.stress_test_failures))
		42	return cls(intersection_api_tests_passed, intersection_stress_test_failures)
		43
		44	@classmethod
		45	def results_from_string(cls, string):
		46	if not string:
		47	return None
		48
		49	try:
		50	parsed_results = json.loads(string)
		51	except ValueError:
		52	_log.error("Invalid JSON results")
		53	return None
		54
		55	if 'allApiTestsPassed' not in parsed_results or 'stressTestFailures' not in parsed_results:
		56	return None
		57
		58	return cls(parsed_results['allApiTestsPassed'], parsed_results['stressTestFailures'])
		59
		60	def equals(self, other):
		61	return (self.all_api_tests_passed == other.all_api_tests_passed and
		62	set(self.stress_test_failures) == set(other.stress_test_failures))
		63
		64	def is_subset(self, other):
65	return set(self.results) <= set(other.results)
66
67	def all_passed(self):
68	return self.all_api_tests_passed and not self.stress_test_failures
69
70	def failing_test_results(self):
71	return self.results
72
73	def test_results(self):
74	return self.results
75
76	def failing_tests(self):
77	return self.results
78
79	# No defined failure limit for JSC.
80	def did_exceed_test_failure_limit(self):
81	return False

Line 0 a/Tools/Scripts/webkitpy/tool/bot/jscews_unittest.py_sec1
- a/Tools/Scripts/webkitpy/tool/bot/jscews_unittest.py +218 lines
		1	# Copyright (C) 2017 Apple Inc. All rights reserved.
		2	#
		3	# Redistribution and use in source and binary forms, with or without
		4	# modification, are permitted provided that the following conditions
		5	# are met:
		6	# 1. Redistributions of source code must retain the above copyright
		7	# notice, this list of conditions and the following disclaimer.
		8	# 2. Redistributions in binary form must reproduce the above copyright
		9	# notice, this list of conditions and the following disclaimer in the
		10	# documentation and/or other materials provided with the distribution.
		11	#
		12	# THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS'' AND
		13	# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
		14	# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
		15	# DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS BE LIABLE FOR
		16	# ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
		17	# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
		18	# SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
		19	# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
		20	# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
		21	# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
		22
		23	import logging
		24	import unittest
		25
		26	from webkitpy.common.net.jsctestresults import JSCTestResults
		27	from webkitpy.common.system.executive import ScriptError
		28	from webkitpy.tool.bot.patchanalysistask import *
		29	from webkitpy.tool.commands.earlywarningsystem import AbstractEarlyWarningSystem
		30	from webkitpy.tool.mocktool import MockTool
		31
		32	_log = logging.getLogger(__name__)
		33
		34
		35	class MockPatchAnalysisTask(PatchAnalysisTask):
		36	def __init__(self, delegate, patch, patches_passed_all_tests):
		37	self._delegate = delegate
		38	self._patch = patch
		39	self._script_error = None
		40	self._results_archive_from_patch_test_run = None
		41	self._results_from_patch_test_run = None
		42	self.error = None
		43	self._patches_passed_all_tests = patches_passed_all_tests
		44	self._test_run_count = 0
		45	self.failure_status_id = 0
		46
		47	def _test(self):
		48	self._test_run_count += 1
		49
		50	if self._patches_passed_all_tests.pop() == True:
		51	return True
		52	self._script_error = ScriptError("Regression test")
		53	return False
		54
		55	def _build_and_test_without_patch(self):
		56	self._test_run_count += 1
		57	return True
		58
		59	def validate(self):
		60	return True
		61
		62	def test_run_count(self):
		63	return self._test_run_count
		64
65
66	# This is the delegate to be used with MockPatchAnalysisTask, above.
67	class MockJSCEarlyWarningSystem(AbstractEarlyWarningSystem):
68	def __init__(self, first_test_results, second_test_results, clean_test_results):
69	AbstractEarlyWarningSystem.__init__(self)
70	self._group = "jsc"
71	self._results_in_order = [clean_test_results, second_test_results, first_test_results]
72
73	def test_results(self):
74	return self._results_in_order.pop()
75
76
77	class JSCEarlyWarningSystemTest(unittest.TestCase):
78	def _results_indicate_all_passed(self, results):
79	if results == None:
80	return False
81	return results.all_passed()
82
83	def _create_task(self, first_test_results, second_test_results, clean_test_results):
84	queue = MockJSCEarlyWarningSystem(first_test_results, second_test_results, clean_test_results)
85	tool = MockTool(log_executive=True)
86	patch = tool.bugs.fetch_attachment(10000)
87	patches_passed_all_tests = map(self._results_indicate_all_passed, [second_test_results, first_test_results])
88	return MockPatchAnalysisTask(queue, patch, patches_passed_all_tests)
89
90	def test_success_case(self):
91	first_test_results = JSCTestResults(True, [])
92	second_test_results = JSCTestResults(True, [])
93	clean_test_results = JSCTestResults(True, [])
94	task = self._create_task(first_test_results, second_test_results, clean_test_results)
95
96	return_value = task._test_patch()
97	self.assertEqual(task.test_run_count(), 1)
98	self.assertTrue(return_value)
99
100	def test_test_failure(self):
101	first_test_results = JSCTestResults(True, ['Fail.js'])
102	second_test_results = JSCTestResults(True, ['Fail.js'])
103	clean_test_results = JSCTestResults(True, [])
104	task = self._create_task(first_test_results, second_test_results, clean_test_results)
105
106	with self.assertRaises(ScriptError):
107	return_value = task._test_patch()
108	self.assertEqual(task.test_run_count(), 3)
109
110	def test_fix(self):
111	first_test_results = JSCTestResults(True, [])
112	second_test_results = JSCTestResults(True, [])
113	clean_test_results = JSCTestResults(True, ['Fail.js'])
114	task = self._create_task(first_test_results, second_test_results, clean_test_results)
115
116	return_value = task._test_patch()
117	self.assertEqual(task.test_run_count(), 1)
118	self.assertTrue(return_value)
119
120	def test_ineffective_patch(self):
121	first_test_results = JSCTestResults(False, ['failure1.js', 'failure2.js'])
122	second_test_results = JSCTestResults(False, ['failure1.js', 'failure2.js'])
123	clean_test_results = JSCTestResults(False, ['failure1.js', 'failure2.js'])
124	task = self._create_task(first_test_results, second_test_results, clean_test_results)
125
126	return_value = task._test_patch()
127	self.assertEqual(task.test_run_count(), 3)
128	self.assertTrue(return_value)
129
130	def test_partially_effective_patch(self):
131	first_test_results = JSCTestResults(True, ['failure2.js'])
132	second_test_results = JSCTestResults(True, ['failure2.js'])
133	clean_test_results = JSCTestResults(False, ['failure1.js', 'failure2.js'])
134	task = self._create_task(first_test_results, second_test_results, clean_test_results)
135
136	return_value = task._test_patch()
137	self.assertEqual(task.test_run_count(), 3)
138	self.assertTrue(return_value)
139
140	def test_different_test_failures_in_patch_and_tree(self):
141	first_test_results = JSCTestResults(False, [])
142	second_test_results = JSCTestResults(False, [])
143	clean_test_results = JSCTestResults(True, ['failure1.js', 'failure2.js'])
144	task = self._create_task(first_test_results, second_test_results, clean_test_results)
145
146	with self.assertRaises(ScriptError):
147	return_value = task._test_patch()
148	self.assertEqual(task.test_run_count(), 3)
149
150	def test_first_results_could_not_be_read(self):
151	first_test_results = None
152	second_test_results = JSCTestResults(True, [])
153	clean_test_results = JSCTestResults(True, [])
154	task = self._create_task(first_test_results, second_test_results, clean_test_results)
155
156	return_value = task._test_patch()
157	self.assertEqual(task.test_run_count(), 1)
158	self.assertFalse(return_value)
159
160	def test_second_results_could_not_be_read(self):
161	first_test_results = JSCTestResults(False, ['failure1.js', 'failure2.js'])
162	second_test_results = None
163	clean_test_results = JSCTestResults(True, [])
164	task = self._create_task(first_test_results, second_test_results, clean_test_results)
165
166	return_value = task._test_patch()
167	self.assertEqual(task.test_run_count(), 2)
168	self.assertFalse(return_value)
169
170	def test_clean_results_could_not_be_read(self):
171	first_test_results = JSCTestResults(True, ['failure2.js'])
172	second_test_results = JSCTestResults(True, ['failure2.js'])
173	clean_test_results = None
174	task = self._create_task(first_test_results, second_test_results, clean_test_results)
175
176	return_value = task._test_patch()
177	self.assertEqual(task.test_run_count(), 3)
178	self.assertFalse(return_value)
179
180	def test_flaky_results_on_clean_tree_pass(self):
181	first_test_results = JSCTestResults(True, ['failure2.js'])
182	second_test_results = JSCTestResults(True, [])
183	clean_test_results = JSCTestResults(True, [])
184	task = self._create_task(first_test_results, second_test_results, clean_test_results)
185
186	return_value = task._test_patch()
187	self.assertTrue(return_value)
188	self.assertEqual(task.test_run_count(), 2)
189
190	def test_flaky_results_on_clean_tree_pass_v2(self):
191	first_test_results = JSCTestResults(True, [])
192	second_test_results = JSCTestResults(True, ['failure2.js'])
193	clean_test_results = JSCTestResults(True, [])
194	task = self._create_task(first_test_results, second_test_results, clean_test_results)
195
196	return_value = task._test_patch()
197	self.assertTrue(return_value)
198	self.assertEqual(task.test_run_count(), 1)
199
200	def test_flaky_results_on_clean_tree_failure(self):
201	first_test_results = JSCTestResults(False, ['failure1.js', 'failure2.js'])
202	second_test_results = JSCTestResults(True, ['failure2.js'])
203	clean_test_results = JSCTestResults(True, [])
204	task = self._create_task(first_test_results, second_test_results, clean_test_results)
205
206	with self.assertRaises(ScriptError):
207	task._test_patch()
208	self.assertEqual(task.test_run_count(), 3)
209
210	def test_flaky_results_on_red_tree_pass(self):
211	first_test_results = JSCTestResults(True, ['failure1.js'])
212	second_test_results = JSCTestResults(True, ['failure1.js', 'failure2.js'])
213	clean_test_results = JSCTestResults(True, ['failure1.js'])
214	task = self._create_task(first_test_results, second_test_results, clean_test_results)
215
216	return_value = task._test_patch()
217	self.assertTrue(return_value)
218	self.assertEqual(task.test_run_count(), 3)

Line 0 a/Tools/Scripts/webkitpy/tool/bot/jsctestresultsreader.py_sec1
- a/Tools/Scripts/webkitpy/tool/bot/jsctestresultsreader.py +47 lines
	1	# Copyright (C) 2017 Apple Inc. All rights reserved.
	2	#
	3	# Redistribution and use in source and binary forms, with or without
	4	# modification, are permitted provided that the following conditions
	5	# are met:
	6	# 1. Redistributions of source code must retain the above copyright
	7	# notice, this list of conditions and the following disclaimer.
	8	# 2. Redistributions in binary form must reproduce the above copyright
	9	# notice, this list of conditions and the following disclaimer in the
	10	# documentation and/or other materials provided with the distribution.
	11	#
	12	# THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS'' AND
	13	# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
	14	# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
	15	# DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS BE LIABLE FOR
	16	# ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
	17	# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
	18	# SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
	19	# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
	20	# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
	21	# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
	22
	23	import logging
	24
	25	from webkitpy.common.net.jsctestresults import JSCTestResults
	26
	27	_log = logging.getLogger(__name__)
	28
	29
	30	class JSCTestResultsReader(object):
	31	def __init__(self, host, results_directory):
	32	self._host = host
	33	self._results_directory = results_directory
	34
	35	def _read_file_contents(self, path):
	36	try:
	37	return self._host.filesystem.read_text_file(path)
	38	except (IOError, KeyError): # File does not exist or can't be read.
	39	return None
	40
	41	def _create_jsc_test_results(self):
	42	results_path = self._host.filesystem.join(self._results_directory, "jsc_test_results.json")
	43	contents = self._read_file_contents(results_path)
	44	return JSCTestResults.results_from_string(contents)
	45
	46	def results(self):
	47	return self._create_jsc_test_results()

Lines 1-4 a/Tools/Scripts/webkitpy/tool/steps/__init__.py_sec1
- a/Tools/Scripts/webkitpy/tool/steps/__init__.py +2 lines
1	# Copyright (C) 2010 Google Inc. All rights reserved.	1	# Copyright (C) 2010 Google Inc. All rights reserved.
		2	# Copyright (C) 2017 Apple Inc. All rights reserved.
2	#	3	#
3	# Redistribution and use in source and binary forms, with or without	4	# Redistribution and use in source and binary forms, with or without
4	# modification, are permitted provided that the following conditions are	5	# modification, are permitted provided that the following conditions are
Lines 33-38 from webkitpy.tool.steps.applypatchwithlocalcommit import ApplyPatchWithLocalCom a/Tools/Scripts/webkitpy/tool/steps/__init__.py_sec2
33	from webkitpy.tool.steps.applywatchlist import ApplyWatchList	34	from webkitpy.tool.steps.applywatchlist import ApplyWatchList
34	from webkitpy.tool.steps.attachtobug import AttachToBug	35	from webkitpy.tool.steps.attachtobug import AttachToBug
35	from webkitpy.tool.steps.build import Build	36	from webkitpy.tool.steps.build import Build
		37	from webkitpy.tool.steps.checkpatchrelevance import CheckPatchRelevance
36	from webkitpy.tool.steps.checkstyle import CheckStyle	38	from webkitpy.tool.steps.checkstyle import CheckStyle
37	from webkitpy.tool.steps.cleanworkingdirectory import CleanWorkingDirectory	39	from webkitpy.tool.steps.cleanworkingdirectory import CleanWorkingDirectory
38	from webkitpy.tool.steps.closebug import CloseBug	40	from webkitpy.tool.steps.closebug import CloseBug

Lines 1-4 a/Tools/Scripts/webkitpy/tool/steps/build.py_sec1
- a/Tools/Scripts/webkitpy/tool/steps/build.py -7 / +15 lines
1	# Copyright (C) 2010 Google Inc. All rights reserved.	1	# Copyright (C) 2010 Google Inc. All rights reserved.
		2	# Copyright (C) 2017 Apple Inc. All rights reserved.
2	#	3	#
3	# Redistribution and use in source and binary forms, with or without	4	# Redistribution and use in source and binary forms, with or without
4	# modification, are permitted provided that the following conditions are	5	# modification, are permitted provided that the following conditions are
Lines 41-68 class Build(AbstractStep): a/Tools/Scripts/webkitpy/tool/steps/build.py_sec2
41	Options.build,	42	Options.build,
42	Options.quiet,	43	Options.quiet,
43	Options.build_style,	44	Options.build_style,
		45	Options.group,
44	]	46	]
45		47
46	def build(self, build_style):	48	def build(self, build_style, group):
47	environment = self._tool.copy_current_environment()	49	environment = self._tool.copy_current_environment()
48	environment.disable_gcc_smartquotes()	50	environment.disable_gcc_smartquotes()
49	environment.disable_jhbuild_VT100_output()	51	environment.disable_jhbuild_VT100_output()
50	env = environment.to_dictionary()	52	env = environment.to_dictionary()
51		53
52	build_webkit_command = self._tool.deprecated_port().build_webkit_command(build_style=build_style)	54	if group == "jsc":
		55	build_command = self._tool.deprecated_port().build_jsc_command(build_style=build_style)
		56	else:
		57	build_command = self._tool.deprecated_port().build_webkit_command(build_style=build_style)
53		58
54	if self._options.architecture:	59	if self._options.architecture:
55	build_webkit_command += ['ARCHS=%s' % self._options.architecture]	60	build_command += ['ARCHS=%s' % self._options.architecture]
56		61
57	self._tool.executive.run_and_throw_if_fail(build_webkit_command, self._options.quiet,	62	self._tool.executive.run_and_throw_if_fail(build_command, self._options.quiet,
58	cwd=self._tool.scm().checkout_root, env=env)	63	cwd=self._tool.scm().checkout_root, env=env)
59		64
60	def run(self, state):	65	def run(self, state):
61	if not self._options.build:	66	if not self._options.build:
62	return	67	return
63	_log.info("Building WebKit")	68	_log.info("Building WebKit")
		69
		70	group = self._options.group
		71
64	if self._options.build_style == "both":	72	if self._options.build_style == "both":
65	self.build("debug")	73	self.build("debug", group)
66	self.build("release")	74	self.build("release", group)
67	else:	75	else:
68	self.build(self._options.build_style)	76	self.build(self._options.build_style, group)

Line 0 a/Tools/Scripts/webkitpy/tool/steps/checkpatchrelevance.py_sec1
- a/Tools/Scripts/webkitpy/tool/steps/checkpatchrelevance.py +75 lines
		1	# Copyright (C) 2017 Apple Inc. All rights reserved.
		2	#
		3	# Redistribution and use in source and binary forms, with or without
		4	# modification, are permitted provided that the following conditions
		5	# are met:
		6	# 1. Redistributions of source code must retain the above copyright
		7	# notice, this list of conditions and the following disclaimer.
		8	# 2. Redistributions in binary form must reproduce the above copyright
		9	# notice, this list of conditions and the following disclaimer in the
		10	# documentation and/or other materials provided with the distribution.
		11	#
		12	# THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS'' AND
		13	# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
		14	# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
		15	# DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS BE LIABLE FOR
		16	# ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
		17	# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
		18	# SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
		19	# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
		20	# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
		21	# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
		22
		23	import logging
		24	import re
		25
		26	from webkitpy.common.system.executive import ScriptError
		27	from webkitpy.tool.steps.abstractstep import AbstractStep
		28	from webkitpy.tool.steps.options import Options
		29
		30	_log = logging.getLogger(__name__)
		31
		32
		33	class CheckPatchRelevance(AbstractStep):
		34	@classmethod
		35	def options(cls):
		36	return AbstractStep.options() + [
		37	Options.group,
		38	]
		39
		40	jsc_paths = [
		41	"JSTests/",
		42	"Source/JavaScriptCore/",
		43	"Source/WTF/"
		44	"Source/bmalloc/",
		45	]
		46
		47	group_to_paths_mapping = {
		48	'jsc': jsc_paths,
		49	}
		50
		51	def _changes_are_relevant(self, changed_files):
		52	# In the default case, all patches are relevant
		53	if self._options.group is None:
		54	return True
		55
		56	patterns = self.group_to_paths_mapping[self._options.group]
		57
		58	for changed_file in changed_files:
		59	for pattern in patterns:
		60	if re.search(pattern, changed_file, re.IGNORECASE):
		61	return True
		62
		63	return False
		64
65	def run(self, state):
66	_log.info("Checking relevance of patch")
67
68	change_list = self._tool.scm().changed_files()
69
70	if self._changes_are_relevant(change_list):
71	_log.info("This patch has relevant changes.")
72	return True
73
74	_log.info("This patch does not have relevant changes.")
75	raise ScriptError(message="This patch does not have relevant changes.")

Lines 1-4 a/Tools/Scripts/webkitpy/tool/steps/options.py_sec1
- a/Tools/Scripts/webkitpy/tool/steps/options.py +2 lines
1	# Copyright (C) 2010 Google Inc. All rights reserved.	1	# Copyright (C) 2010 Google Inc. All rights reserved.
		2	# Copyright (C) 2017 Apple Inc. All rights reserved.
2	#	3	#
3	# Redistribution and use in source and binary forms, with or without	4	# Redistribution and use in source and binary forms, with or without
4	# modification, are permitted provided that the following conditions are	5	# modification, are permitted provided that the following conditions are
Lines 46-51 class Options(object): a/Tools/Scripts/webkitpy/tool/steps/options.py_sec2
46	email = make_option("--email", action="store", type="string", dest="email", help="Email address to use in ChangeLogs.")	47	email = make_option("--email", action="store", type="string", dest="email", help="Email address to use in ChangeLogs.")
47	force_clean = make_option("--force-clean", action="store_true", dest="force_clean", default=False, help="Clean working directory before applying patches (removes local changes and commits)")	48	force_clean = make_option("--force-clean", action="store_true", dest="force_clean", default=False, help="Clean working directory before applying patches (removes local changes and commits)")
48	git_commit = make_option("-g", "--git-commit", action="store", dest="git_commit", help="Operate on a local commit. If a range, the commits are squashed into one. <ref>.... includes the working copy changes. UPSTREAM can be used for the upstream/tracking branch.")	49	git_commit = make_option("-g", "--git-commit", action="store", dest="git_commit", help="Operate on a local commit. If a range, the commits are squashed into one. <ref>.... includes the working copy changes. UPSTREAM can be used for the upstream/tracking branch.")
		50	group = make_option("--group", action="store", dest="group", default=None, help="")
49	local_commit = make_option("--local-commit", action="store_true", dest="local_commit", default=False, help="Make a local commit for each applied patch")	51	local_commit = make_option("--local-commit", action="store_true", dest="local_commit", default=False, help="Make a local commit for each applied patch")
50	non_interactive = make_option("--non-interactive", action="store_true", dest="non_interactive", default=False, help="Never prompt the user, fail as fast as possible.")	52	non_interactive = make_option("--non-interactive", action="store_true", dest="non_interactive", default=False, help="Never prompt the user, fail as fast as possible.")
51	obsolete_patches = make_option("--no-obsolete", action="store_false", dest="obsolete_patches", default=True, help="Do not obsolete old patches before posting this one.")	53	obsolete_patches = make_option("--no-obsolete", action="store_false", dest="obsolete_patches", default=True, help="Do not obsolete old patches before posting this one.")

Lines 1-4 a/Tools/Scripts/webkitpy/tool/steps/steps_unittest.py_sec1
- a/Tools/Scripts/webkitpy/tool/steps/steps_unittest.py -2 / +106 lines
1	# Copyright (C) 2010 Google Inc. All rights reserved.	1	# Copyright (C) 2010 Google Inc. All rights reserved.
		2	# Copyright (C) 2017 Apple Inc. All rights reserved.
2	#	3	#
3	# Redistribution and use in source and binary forms, with or without	4	# Redistribution and use in source and binary forms, with or without
4	# modification, are permitted provided that the following conditions are	5	# modification, are permitted provided that the following conditions are
Lines 28-33 a/Tools/Scripts/webkitpy/tool/steps/steps_unittest.py_sec2
28		29
29	import unittest	30	import unittest
30		31
		32	from webkitpy.common.system.executive import ScriptError
31	from webkitpy.common.system.outputcapture import OutputCapture	33	from webkitpy.common.system.outputcapture import OutputCapture
32	from webkitpy.common.config.ports import DeprecatedPort	34	from webkitpy.common.config.ports import DeprecatedPort
33	from webkitpy.tool.mocktool import MockOptions, MockTool	35	from webkitpy.tool.mocktool import MockOptions, MockTool
Lines 38-43 from webkitpy.tool import steps a/Tools/Scripts/webkitpy/tool/steps/steps_unittest.py_sec3
38	class StepsTest(unittest.TestCase):	40	class StepsTest(unittest.TestCase):
39	def _step_options(self):	41	def _step_options(self):
40	options = MockOptions()	42	options = MockOptions()
		43	options.group = None
41	options.non_interactive = True	44	options.non_interactive = True
42	options.port = 'MOCK port'	45	options.port = 'MOCK port'
43	options.quiet = True	46	options.quiet = True
Lines 111-117 MOCK run_and_throw_if_fail: ['Tools/Scripts/test-webkitpy'], cwd=/mock-checkout a/Tools/Scripts/webkitpy/tool/steps/steps_unittest.py_sec4
111	Running Perl unit tests	114	Running Perl unit tests
112	MOCK run_and_throw_if_fail: ['Tools/Scripts/test-webkitperl'], cwd=/mock-checkout	115	MOCK run_and_throw_if_fail: ['Tools/Scripts/test-webkitperl'], cwd=/mock-checkout
113	Running JavaScriptCore tests	116	Running JavaScriptCore tests
114	MOCK run_and_throw_if_fail: ['Tools/Scripts/run-javascriptcore-tests'], cwd=/mock-checkout	117	MOCK run_and_throw_if_fail: ['Tools/Scripts/run-javascriptcore-tests', '--no-fail-fast', '--json-output=jsc_test_results.json'], cwd=/mock-checkout
115	Running bindings generation tests	118	Running bindings generation tests
116	MOCK run_and_throw_if_fail: ['Tools/Scripts/run-bindings-tests'], cwd=/mock-checkout	119	MOCK run_and_throw_if_fail: ['Tools/Scripts/run-bindings-tests'], cwd=/mock-checkout
117	Running run-webkit-tests	120	Running run-webkit-tests
Lines 133-142 MOCK run_and_throw_if_fail: ['Tools/Scripts/test-webkitpy'], cwd=/mock-checkout a/Tools/Scripts/webkitpy/tool/steps/steps_unittest.py_sec5
133	Running Perl unit tests	136	Running Perl unit tests
134	MOCK run_and_throw_if_fail: ['Tools/Scripts/test-webkitperl'], cwd=/mock-checkout	137	MOCK run_and_throw_if_fail: ['Tools/Scripts/test-webkitperl'], cwd=/mock-checkout
135	Running JavaScriptCore tests	138	Running JavaScriptCore tests
136	MOCK run_and_throw_if_fail: ['Tools/Scripts/run-javascriptcore-tests'], cwd=/mock-checkout	139	MOCK run_and_throw_if_fail: ['Tools/Scripts/run-javascriptcore-tests', '--no-fail-fast', '--json-output=jsc_test_results.json'], cwd=/mock-checkout
137	Running bindings generation tests	140	Running bindings generation tests
138	MOCK run_and_throw_if_fail: ['Tools/Scripts/run-bindings-tests'], cwd=/mock-checkout	141	MOCK run_and_throw_if_fail: ['Tools/Scripts/run-bindings-tests'], cwd=/mock-checkout
139	Running run-webkit-tests	142	Running run-webkit-tests
140	MOCK run_and_throw_if_fail: ['Tools/Scripts/run-webkit-tests', '--debug', '--quiet'], cwd=/mock-checkout	143	MOCK run_and_throw_if_fail: ['Tools/Scripts/run-webkit-tests', '--debug', '--quiet'], cwd=/mock-checkout
141	"""	144	"""
142	OutputCapture().assert_outputs(self, step.run, [{}], expected_logs=expected_logs)	145	OutputCapture().assert_outputs(self, step.run, [{}], expected_logs=expected_logs)
		146
		147	def test_runtests_jsc(self):
		148	mock_options = self._step_options()
		149	mock_options.non_interactive = False
		150	mock_options.build_style = "release"
		151	mock_options.group = "jsc"
		152	step = steps.RunTests(MockTool(log_executive=True), mock_options)
		153	tool = MockTool(log_executive=True)
		154	# FIXME: We shouldn't use a real port-object here, but there is too much to mock at the moment.
		155	tool._deprecated_port = DeprecatedPort()
		156	step = steps.RunTests(tool, mock_options)
		157	expected_logs = """MOCK run_and_throw_if_fail: ['Tools/Scripts/run-javascriptcore-tests', '--no-fail-fast', '--json-output=jsc_test_results.json'], cwd=/mock-checkout
		158	"""
		159	OutputCapture().assert_outputs(self, step.run, [{}], expected_logs=expected_logs)
		160
		161	def test_runtests_jsc_debug(self):
		162	mock_options = self._step_options()
		163	mock_options.non_interactive = False
		164	mock_options.build_style = "debug"
		165	mock_options.group = "jsc"
		166	tool = MockTool(log_executive=True)
		167	# FIXME: We shouldn't use a real port-object here, but there is too much to mock at the moment.
		168	tool._deprecated_port = DeprecatedPort()
		169	step = steps.RunTests(tool, mock_options)
		170	expected_logs = """MOCK run_and_throw_if_fail: ['Tools/Scripts/run-javascriptcore-tests', '--no-fail-fast', '--json-output=jsc_test_results.json', '--debug'], cwd=/mock-checkout
		171	"""
		172	OutputCapture().assert_outputs(self, step.run, [{}], expected_logs=expected_logs)
		173
		174	def test_build_jsc_debug(self):
		175	mock_options = self._step_options()
		176	mock_options.non_interactive = False
		177	mock_options.build_style = "debug"
		178	mock_options.build = True
		179	mock_options.architecture = True
		180	mock_options.group = "jsc"
		181	tool = MockTool(log_executive=True)
		182	# FIXME: We shouldn't use a real port-object here, but there is too much to mock at the moment.
		183	tool._deprecated_port = DeprecatedPort()
		184	step = steps.Build(tool, mock_options)
		185	expected_logs = """Building WebKit
		186	MOCK run_and_throw_if_fail: ['Tools/Scripts/build-jsc', '--debug', 'ARCHS=True'], cwd=/mock-checkout, env={'LC_ALL': 'C', 'TERM': 'none', 'MOCK_ENVIRON_COPY': '1'}
		187	"""
		188	OutputCapture().assert_outputs(self, step.run, [{}], expected_logs=expected_logs)
		189
		190	def test_build_jsc(self):
		191	mock_options = self._step_options()
		192	mock_options.non_interactive = False
		193	mock_options.build_style = "release"
		194	mock_options.build = True
		195	mock_options.architecture = True
		196	mock_options.group = "jsc"
		197	tool = MockTool(log_executive=True)
		198	# FIXME: We shouldn't use a real port-object here, but there is too much to mock at the moment.
		199	tool._deprecated_port = DeprecatedPort()
		200	step = steps.Build(tool, mock_options)
		201	expected_logs = """Building WebKit
		202	MOCK run_and_throw_if_fail: ['Tools/Scripts/build-jsc', '--release', 'ARCHS=True'], cwd=/mock-checkout, env={'LC_ALL': 'C', 'TERM': 'none', 'MOCK_ENVIRON_COPY': '1'}
		203	"""
		204	OutputCapture().assert_outputs(self, step.run, [{}], expected_logs=expected_logs)
		205
		206	def test_patch_relevant(self):
		207	self.maxDiff = None
		208	mock_options = self._step_options()
		209	tool = MockTool(log_executive=True)
210	tool.scm()._mockChangedFiles = ["JSTests/MockFile1", "ChangeLog"]
211	# FIXME: We shouldn't use a real port-object here, but there is too much to mock at the moment.
212	tool._deprecated_port = DeprecatedPort()
213	step = steps.CheckPatchRelevance(tool, mock_options)
214	expected_logs = """Checking relevance of patch
215	This patch has relevant changes.
216	"""
217	OutputCapture().assert_outputs(self, step.run, [{}], expected_logs=expected_logs)
218
219	def test_patch_relevant_jsc(self):
220	self.maxDiff = None
221	mock_options = self._step_options()
222	mock_options.group = "jsc"
223	tool = MockTool(log_executive=True)
224	tool.scm()._mockChangedFiles = ["JSTests/MockFile1", "ChangeLog"]
225	# FIXME: We shouldn't use a real port-object here, but there is too much to mock at the moment.
226	tool._deprecated_port = DeprecatedPort()
227	step = steps.CheckPatchRelevance(tool, mock_options)
228	expected_logs = """Checking relevance of patch
229	This patch has relevant changes.
230	"""
231	OutputCapture().assert_outputs(self, step.run, [{}], expected_logs=expected_logs)
232
233	def test_patch_not_relevant_jsc(self):
234	self.maxDiff = None
235	mock_options = self._step_options()
236	mock_options.group = "jsc"
237	tool = MockTool(log_executive=True)
238	tool.scm()._mockChangedFiles = ["Tools/ChangeLog", "Tools/Scripts/webkitpy/tool/steps/steps_unittest.py"]
239	# FIXME: We shouldn't use a real port-object here, but there is too much to mock at the moment.
240	tool._deprecated_port = DeprecatedPort()
241	step = steps.CheckPatchRelevance(tool, mock_options)
242	expected_logs = """Checking relevance of patch
243	This patch does not have relevant changes.
244	"""
245	with self.assertRaises(ScriptError):
246	OutputCapture().assert_outputs(self, step.run, [{}], expected_logs=expected_logs)