Attachment #298802 for bug #162458

View Details Raw Unified Return to Bug 162458

| Differences between



2017-01-13  Srinivasan Vijayaraghavan  <svijayaraghavan@apple.com>

        EWS should run JavaScriptCore tests
        https://bugs.webkit.org/show_bug.cgi?id=162458

        Reviewed by NOBODY (OOPS!).

        * Scripts/webkitpy/common/config/ews.json: Add config info for JSC EWS.
        * Scripts/webkitpy/common/config/ports.py: Add support for JSC EWS in Mac Port.
        (DeprecatedPort.build_jsc_command): Added - command to build JSC only (quicker than building all of WebKit).
        (DeprecatedPort.run_javascriptcore_tests_command): Allow JSC EWS to only run JSC tests.
        (MacPort.run_webkit_tests_command): Check for JSC.
        * Scripts/webkitpy/common/net/jsctestresults.py: Added.
        (JSCTestResults):
        (JSCTestResults.results_from_string): Use json library to parse results.
        (JSCTestResults.__init__): Initialize members.
        (JSCTestResults.equals): This enables unit testing.
        (JSCTestResults.is_subset): Checks if one set of failures is a subset of another.
        (JSCTestResults.stress_test_failures): Getter.
        (JSCTestResults.all_api_tests_passed): Getter.
        (JSCTestResults.all_passed): Getter.
        (JSCTestResults.failing_test_results): Getter. Some of these exist for interopability reasons.
        (JSCTestResults.test_results): Getter.
        (JSCTestResults.failing_tests): Getter.
        (JSCTestResults.did_exceed_test_failure_limit): Getter.
        * Scripts/webkitpy/common/net/jsctestresults_unittest.py: Added.
        (JSCTestResultsTest): Class to test JSCTestResults.
        (JSCTestResultsTest.test_results_from_string): Tests parsing.
       * Scripts/webkitpy/tool/bot/jscews_unittest.py: Added.
        (MockPatchAnalysisTask): Mocked-out version of PatchAnalysisTask that doesn't run commands.
        (MockPatchAnalysisTask.__init__): Sets attributes.
        (MockPatchAnalysisTask._test): Override to not run command.
        (MockPatchAnalysisTask._build_and_test_without_patch): Override to not run command.
        (MockPatchAnalysisTask.validate): Assume mocked patch is valid for purposes of testing retry logic.
        (MockPatchAnalysisTask.test_run_count): Specific to the mocked version, to test retry logic.
        (MockJSCEarlyWarningSystem): Mocked-out version of AbstarctEarlyWarningSystem so we can provide test results.
        (MockJSCEarlyWarningSystem.__init__): Sets attributes, also sets group to jsc.
        (MockJSCEarlyWarningSystem.test_results): Returns test results provided by us, instead of using a JSON reader.
        (JSCEarlyWarningSystemTest): Class to test retry logic in below situations.
        (JSCEarlyWarningSystemTest._create_task): Helper function to abstract out common code.
        (JSCEarlyWarningSystemTest.test_success_case): Clean patch on clean tree.
        (JSCEarlyWarningSystemTest.test_test_failure): Failed patch on clean tree.
        (JSCEarlyWarningSystemTest.test_fix): Patch that fixes all tree redness.
        (JSCEarlyWarningSystemTest.test_ineffective_patch): Patch that has same failures as tree.
        (JSCEarlyWarningSystemTest.test_partially_effective_patch): Patch fixes some failures but adds no new failures.
        (JSCEarlyWarningSystemTest.test_different_test_failures): Patch that has some failures not present in tree.
        (JSCEarlyWarningSystemTest.test_first_results_could_not_be_read): Results json file from patch could not be read.
        (JSCEarlyWarningSystemTest.test_clean_results_could_not_be_read): Results json file from clean tree could not be read.
        * Scripts/webkitpy/tool/bot/jsctestresultsreader.py: Added.
        (JSCTestResultsReader): Reads results file.
        (JSCTestResultsReader.__init__): Sets attributes.
        (JSCTestResultsReader._read_file_contents): Reads file.
        (JSCTestResultsReader._create_jsc_test_results): Creates a JSCTestResults object.
        (JSCTestResultsReader.results): Invokes _create_jsc_test_results to return a JSCTestResults object.
        * Scripts/webkitpy/tool/bot/patchanalysistask.py:
        (PatchAnalysisTask._build): Check for JSC.
        (PatchAnalysisTask._build_without_patch): Check for JSC.
        (PatchAnalysisTask._test): Check for JSC.
        (PatchAnalysisTask._build_and_test_without_patch): Check for JSC.
        (PatchAnalysisTask._test_patch): Add retry logic for JSC.
        * Scripts/webkitpy/tool/commands/earlywarningsystem.py:
        (AbstractEarlyWarningSystem.begin_work_queue): Use JSCTestResultsReader instead of LayoutTestResultsReader in JSC EWS.
        (AbstractEarlyWarningSystem.test_results): _layout_test_results_reader -> _test_results_reader.
        (AbstractEarlyWarningSystem.archive_last_test_results): _layout_test_results_reader -> _test_results_reader.
        (AbstractEarlyWarningSystem.group): This attribute determines the type of EWS (eg. JSC).
        (AbstractEarlyWarningSystem.load_ews_classes): Add _group.
        * Scripts/webkitpy/tool/commands/earlywarningsystem_unittest.py:
        (EarlyWarningSystemTest._default_expected_logs): Add _group.
        * Scripts/webkitpy/tool/steps/build.py:
        (Build.options): Check for JSC.
        (Build.build): Check for JSC.
        (Build.run): Check for JSC.
        * Scripts/webkitpy/tool/steps/options.py:
        (Options): Add --group command line option.
        * Scripts/webkitpy/tool/steps/runtests.py:
        (RunTests.options): Add group.
        (RunTests.run): Check for JSC.
        (RunTests._run_webkit_tests): Check for JSC.
        (RunTests): Add _group attribute.
        (RunTests._run_javascriptcore_tests): New.
        * Scripts/webkitpy/tool/steps/steps_unittest.py: Change expected command for running jsc tests.

2016-09-25  Gyuyoung Kim  <gyuyoung.kim@webkit.org>

        [EFL] Adjust EFL indentation to EFL MiniBrowser

Lines 50-54 a/Tools/Scripts/webkitpy/common/config/ews.json_sec1
- a/Tools/Scripts/webkitpy/common/config/ews.json +6 lines
50	"port": "mac",	50	"port": "mac",
51	"name": "mac-32bit-ews",	51	"name": "mac-32bit-ews",
52	"architecture": "i386"	52	"architecture": "i386"
		53	},
		54	"JSC EWS": {
		55	"port": "mac",
		56	"name": "jsc-ews",
		57	"group": "jsc",
		58	"runTests": true
53	}	59	}
54	}	60	}



# Copyright (C) 2009, Google Inc. All rights reserved.
# Copyright (C) 2013 Nokia Corporation and/or its subsidiary(-ies).
# Copyright (C) 2015, 2017 Apple Inc. All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are

            command.append("--release")
        return command

    def build_jsc_command(self, build_style=None):
        command = self.script_shell_command("build-jsc")
        if build_style == "debug":
            command.append("--debug")
        if build_style == "release":
            command.append("--release")
        return command

    def run_javascriptcore_tests_command(self, build_style=None):
        command = self.script_shell_command("run-javascriptcore-tests")
        command.append("--no-fail-fast")
        command.append("--json-output=jsc_test_results.json")
        if (build_style == "debug"):
            command.append("--debug")
        return command

    def run_webkit_tests_command(self, build_style=None):
        command = self.script_shell_command("run-webkit-tests")

class MacPort(DeprecatedPort):
    port_flag_name = "mac"

    def run_webkit_tests_command(self, build_style=None, test_suite=None):
        if (test_suite == "jsc"):
            return super(MacPort, self).run_javascriptcore_tests_command(build_style)

        command = super(MacPort, self).run_webkit_tests_command(build_style)
        command.append("--dump-render-tree")
        return command

Lines 1-4 a/Tools/Scripts/webkitpy/common/config/ports_unittest.py_sec1
- a/Tools/Scripts/webkitpy/common/config/ports_unittest.py +5 lines
1	# Copyright (c) 2009, Google Inc. All rights reserved.	1	# Copyright (c) 2009, Google Inc. All rights reserved.
		2	# Copyright (C) 2017 Apple Inc. All rights reserved.
2	#	3	#
3	# Redistribution and use in source and binary forms, with or without	4	# Redistribution and use in source and binary forms, with or without
4	# modification, are permitted provided that the following conditions are	5	# modification, are permitted provided that the following conditions are
Lines 39-44 class DeprecatedPortTest(unittest.TestCase): a/Tools/Scripts/webkitpy/common/config/ports_unittest.py_sec2
39	self.assertEqual(MacPort().build_webkit_command(build_style="debug"), DeprecatedPort().script_shell_command("build-webkit") + ["--debug"])	40	self.assertEqual(MacPort().build_webkit_command(build_style="debug"), DeprecatedPort().script_shell_command("build-webkit") + ["--debug"])
40	self.assertEqual(MacPort().build_webkit_command(build_style="release"), DeprecatedPort().script_shell_command("build-webkit") + ["--release"])	41	self.assertEqual(MacPort().build_webkit_command(build_style="release"), DeprecatedPort().script_shell_command("build-webkit") + ["--release"])
41		42
		43	self.assertEqual(MacPort().build_jsc_command(), DeprecatedPort().script_shell_command("build-jsc"))
		44	self.assertEqual(MacPort().build_jsc_command(build_style="release"), DeprecatedPort().script_shell_command("build-jsc") + ["--release"])
		45	self.assertEqual(MacPort().build_jsc_command(build_style="debug"), DeprecatedPort().script_shell_command("build-jsc") + ["--debug"])
		46
42	def test_gtk_wk2_port(self):	47	def test_gtk_wk2_port(self):
43	self.assertEqual(GtkWK2Port().flag(), "--port=gtk-wk2")	48	self.assertEqual(GtkWK2Port().flag(), "--port=gtk-wk2")
44	self.assertEqual(GtkWK2Port().run_webkit_tests_command(), DeprecatedPort().script_shell_command("run-webkit-tests") + ["--gtk"])	49	self.assertEqual(GtkWK2Port().run_webkit_tests_command(), DeprecatedPort().script_shell_command("run-webkit-tests") + ["--gtk"])



# Copyright (C) 2017 Apple Inc. All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions
# are met:
# 1.  Redistributions of source code must retain the above copyright
#     notice, this list of conditions and the following disclaimer.
# 2.  Redistributions in binary form must reproduce the above copyright
#     notice, this list of conditions and the following disclaimer in the
#     documentation and/or other materials provided with the distribution.
#
# THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS'' AND
# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
# DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS BE LIABLE FOR
# ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
# SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

import json
import logging

_log = logging.getLogger(__name__)


class JSCTestResults(object):
    @classmethod
    def results_from_string(cls, string):
        if not string:
            return None

        try:
            parsed_results = json.loads(string)
        except ValueError:
            return None

        return cls(parsed_results['allApiTestsPassed'], parsed_results['stressTestFailures'])

    def __init__(self, all_api_tests_passed, stress_test_failures):
        self._all_api_tests_passed = all_api_tests_passed
        self._stress_test_failures = stress_test_failures

        self._test_results = stress_test_failures[:]
        if not self._all_api_tests_passed:
            self._test_results.append("apiTests")

    def equals(self, other):
        return self._all_api_tests_passed == other._all_api_tests_passed and set(self._stress_test_failures) == set(other._stress_test_failures)

    def is_subset(self, other):
        return set(self.test_results()) <= set(other.test_results())

    def stress_test_failures(self):
        return self._stress_test_failures

    def all_api_tests_passed(self):
        return self._all_api_tests_passed

    def all_passed(self):
        return self._all_api_tests_passed and not self._stress_test_failures

    def failing_test_results(self):
        return self.stress_test_failures()

    def test_results(self):
        return self._test_results

    def failing_tests(self):
        return self._test_results

    # No defined failure limit for JSC.
    def did_exceed_test_failure_limit(self):
        return False



# Copyright (C) 2017 Apple Inc. All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions
# are met:
# 1.  Redistributions of source code must retain the above copyright
#     notice, this list of conditions and the following disclaimer.
# 2.  Redistributions in binary form must reproduce the above copyright
#     notice, this list of conditions and the following disclaimer in the
#     documentation and/or other materials provided with the distribution.
#
# THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS'' AND
# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
# DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS BE LIABLE FOR
# ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
# SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

import unittest

from webkitpy.common.net.jsctestresults import JSCTestResults


class JSCTestResultsTest(unittest.TestCase):
    def test_results_from_string(self):
        no_failures_string = '{"allApiTestsPassed": true, "stressTestFailures":[]}'
        no_failures_results = JSCTestResults(True, [])
        self.assertTrue(no_failures_results.equals(JSCTestResults.results_from_string(no_failures_string)))

        api_test_failures_string = '{"allApiTestsPassed": false, "stressTestFailures":[]}'
        api_test_failures_results = JSCTestResults(False, [])
        self.assertTrue(api_test_failures_results.equals(JSCTestResults.results_from_string(api_test_failures_string)))

        many_failures_string = '{"allApiTestsPassed": false, "stressTestFailures":["es6.yaml/es6/typed_arrays_Int16Array.js.default", "es6.yaml/es6/typed_arrays_Int8Array.js.default"]}'
        many_failures_results = JSCTestResults(False, ["es6.yaml/es6/typed_arrays_Int16Array.js.default", "es6.yaml/es6/typed_arrays_Int8Array.js.default"])
        self.assertTrue(many_failures_results.equals(JSCTestResults.results_from_string(many_failures_string)))

        self.assertFalse(no_failures_results == api_test_failures_results)
        self.assertFalse(api_test_failures_results == many_failures_results)



# Copyright (C) 2017 Apple Inc. All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions
# are met:
# 1.  Redistributions of source code must retain the above copyright
#     notice, this list of conditions and the following disclaimer.
# 2.  Redistributions in binary form must reproduce the above copyright
#     notice, this list of conditions and the following disclaimer in the
#     documentation and/or other materials provided with the distribution.
#
# THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS'' AND
# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
# DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS BE LIABLE FOR
# ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
# SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

import logging
import unittest

from webkitpy.common.net.jsctestresults import JSCTestResults
from webkitpy.common.system.executive import ScriptError
from webkitpy.tool.bot.patchanalysistask import *
from webkitpy.tool.commands.earlywarningsystem import AbstractEarlyWarningSystem
from webkitpy.tool.mocktool import MockTool

_log = logging.getLogger(__name__)


class MockPatchAnalysisTask(PatchAnalysisTask):
    def __init__(self, delegate, patch, first_test_all_passed):
        self._delegate = delegate
        self._patch = patch
        self._script_error = None
        self._results_archive_from_patch_test_run = None
        self._results_from_patch_test_run = None
        self.error = None
        self._first_test_all_passed = first_test_all_passed
        self._test_run_count = 0

    def _test(self):
        self._test_run_count += 1

        if self._first_test_all_passed == True:
            return True
        else:
            self._script_error = ScriptError("Regression test")
            return False

    def _build_and_test_without_patch(self):
        self._test_run_count += 1
        return True

    def validate(self):
        return True

    def test_run_count(self):
        return self._test_run_count


# This is the delegate to be used with MockPatchAnalysisTask, above.
class MockJSCEarlyWarningSystem(AbstractEarlyWarningSystem):
    def __init__(self, first_test_results, clean_test_results):
        AbstractEarlyWarningSystem.__init__(self)
        self._group = "jsc"
        self._results_in_order = [clean_test_results, first_test_results]

    def test_results(self):
        return self._results_in_order.pop()


class JSCEarlyWarningSystemTest(unittest.TestCase):
    def _create_task(self, first_test_results, clean_test_results):
        queue = MockJSCEarlyWarningSystem(first_test_results, clean_test_results)
        tool = MockTool(log_executive=True)
        patch = tool.bugs.fetch_attachment(10000)
        patch_passed_all_tests = False if first_test_results is None else first_test_results.all_passed()
        return MockPatchAnalysisTask(queue, patch, patch_passed_all_tests)

    def test_success_case(self):
        first_test_results = JSCTestResults(True, [])
        clean_test_results = JSCTestResults(True, [])
        task = self._create_task(first_test_results, clean_test_results)

        return_value = task._test_patch()
        self.assertEqual(task.test_run_count(), 1)
        self.assertEqual(return_value, True)

    def test_test_failure(self):
        first_test_results = JSCTestResults(True, ['Fail.js'])
        clean_test_results = JSCTestResults(True, [])
        task = self._create_task(first_test_results, clean_test_results)

        with self.assertRaises(ScriptError):
            return_value = task._test_patch()
        self.assertEqual(task.test_run_count(), 2)

    def test_fix(self):
        first_test_results = JSCTestResults(True, [])
        clean_test_results = JSCTestResults(True, ['Fail.js'])
        task = self._create_task(first_test_results, clean_test_results)

        return_value = task._test_patch()
        self.assertEqual(task.test_run_count(), 1)
        self.assertEqual(return_value, True)

    def test_ineffective_patch(self):
        first_test_results = JSCTestResults(False, ['failure1.js', 'failure2.js'])
        clean_test_results = JSCTestResults(False, ['failure1.js', 'failure2.js'])
        task = self._create_task(first_test_results, clean_test_results)

        return_value = task._test_patch()
        self.assertEqual(task.test_run_count(), 2)
        self.assertEqual(return_value, True)

    def test_partially_effective_patch(self):
        first_test_results = JSCTestResults(True, ['failure2.js'])
        clean_test_results = JSCTestResults(False, ['failure1.js', 'failure2.js'])
        task = self._create_task(first_test_results, clean_test_results)

        return_value = task._test_patch()
        self.assertEqual(task.test_run_count(), 2)
        self.assertEqual(return_value, True)

    def test_different_test_failures(self):
        first_test_results = JSCTestResults(False, [])
        clean_test_results = JSCTestResults(True, ['failure1.js', 'failure2.js'])
        task = self._create_task(first_test_results, clean_test_results)

        with self.assertRaises(ScriptError):
            return_value = task._test_patch()
        self.assertEqual(task.test_run_count(), 2)

    def test_first_results_could_not_be_read(self):
        first_test_results = None
        clean_test_results = JSCTestResults(True, [])
        task = self._create_task(first_test_results, clean_test_results)

        with self.assertRaises(ScriptError):
            return_value = task._test_patch()
        self.assertEqual(task.test_run_count(), 1)

    def test_clean_results_could_not_be_read(self):
        first_test_results = JSCTestResults(True, ['failure2.js'])
        clean_test_results = None
        task = self._create_task(first_test_results, clean_test_results)

        with self.assertRaises(ScriptError):
            return_value = task._test_patch()
        self.assertEqual(task.test_run_count(), 2)



# Copyright (C) 2017 Apple Inc. All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions
# are met:
# 1.  Redistributions of source code must retain the above copyright
#     notice, this list of conditions and the following disclaimer.
# 2.  Redistributions in binary form must reproduce the above copyright
#     notice, this list of conditions and the following disclaimer in the
#     documentation and/or other materials provided with the distribution.
#
# THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS'' AND
# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
# DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS BE LIABLE FOR
# ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
# SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

import logging

from webkitpy.common.net.jsctestresults import JSCTestResults

_log = logging.getLogger(__name__)


class JSCTestResultsReader(object):
    def __init__(self, host, results_directory):
        self._host = host
        self._results_directory = results_directory

    def _read_file_contents(self, path):
        try:
            return self._host.filesystem.read_text_file(path)
        except IOError, e:  # File does not exist or can't be read.
            return None

    def _create_jsc_test_results(self):
        results_path = self._host.filesystem.join(self._results_directory, "jsc_test_results.json")
        contents = self._read_file_contents(results_path)
        return JSCTestResults.results_from_string(contents)

    def results(self):
        return self._create_jsc_test_results()



# Copyright (c) 2010 Google Inc. All rights reserved.
# Copyright (C) 2017 Apple Inc. All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are

        "Patch does not apply")

    def _build(self):
        args = [
            "build",
            "--no-clean",
            "--no-update",
            "--build-style=%s" % self._delegate.build_style(),
        ]

        if hasattr(self._delegate, 'group'):
            args.append("--group=%s" % self._delegate.group())

        return self._run_command(
            args,
            "Built patch",
            "Patch does not build")

    def _build_without_patch(self):
        args = [
            "build",
            "--force-clean",
            "--no-update",
            "--build-style=%s" % self._delegate.build_style(),
        ]

        if hasattr(self._delegate, 'group'):
            args.append("--group=%s" % self._delegate.group())

        return self._run_command(
            args,
            "Able to build without patch",
            "Unable to build without patch")

    def _test(self):
        args = [
            "build-and-test",
            "--no-clean",
            "--no-update",

            "--test",
            "--non-interactive",
            "--build-style=%s" % self._delegate.build_style(),
        ]

        if hasattr(self._delegate, 'group'):
            args.append("--group=%s" % self._delegate.group())

        return self._run_command(
            args,
            "Passed tests",
            "Patch does not pass tests")

    def _build_and_test_without_patch(self):
        args = [
            "build-and-test",
            "--force-clean",
            "--no-update",

            "--test",
            "--non-interactive",
            "--build-style=%s" % self._delegate.build_style(),
        ]

        if hasattr(self._delegate, 'group'):
            args.append("--group=%s" % self._delegate.group())

        return self._run_command(
            args,
            "Able to pass tests without patch",
            "Unable to pass tests without patch (tree is red?)")

    def _land(self):
        # Unclear if this should pass --quiet or not.  If --parent-command always does the reporting, then it should.

        if self._test():
            return True

        # FIXME: Abstract out the retry logic instead of casing on the test suite here.
        if hasattr(self._delegate, 'group') and self._delegate.group() == "jsc":
            first_results = self._delegate.test_results()
            first_script_error = self._script_error

            if first_results is None:
                return self.report_failure(None, None, ScriptError("Could not read JSC test results for patch."))
        if self._test() and not first_results.did_exceed_test_failure_limit():
            # Only report flaky tests if we were successful at parsing results.json and archiving results.
            if first_results and first_results_archive:
                self._report_flaky_tests(first_results.failing_test_results(), first_results_archive)
            return True

            if first_results.all_passed():
                return True
        second_script_error = self._script_error
        second_failure_status_id = self.failure_status_id

        if second_results.did_exceed_test_failure_limit() and first_results.did_exceed_test_failure_limit():
            self._build_and_test_without_patch()
            clean_tree_results = self._delegate.test_results()

            if clean_tree_results is None:
                return self.report_failure(None, None, ScriptError("Could not read JSC test results for clean tree."))

            if first_results.is_subset(clean_tree_results):
                return True

            return self.report_failure(None, first_results, first_script_error)

        else:
            # Note: archive_last_test_results deletes the results directory, making these calls order-sensitve.
            # We could remove this dependency by building the test_results from the archive.
            first_results = self._delegate.test_results()
            first_results_archive = self._delegate.archive_last_test_results(self._patch)
            first_script_error = self._script_error
            first_failure_status_id = self.failure_status_id

            if self._test() and not first_results.did_exceed_test_failure_limit():
                # Only report flaky tests if we were successful at parsing results.json and archiving results.
                if first_results and first_results_archive:
                    self._report_flaky_tests(first_results.failing_test_results(), first_results_archive)
                return True

            second_results = self._delegate.test_results()
            second_results_archive = self._delegate.archive_last_test_results(self._patch)
            second_script_error = self._script_error
            second_failure_status_id = self.failure_status_id

            if second_results.did_exceed_test_failure_limit() and first_results.did_exceed_test_failure_limit():
                self._build_and_test_without_patch()
                clean_tree_results = self._delegate.test_results()

                if (len(first_results.failing_tests()) - len(clean_tree_results.failing_tests())) <= 5:
                    return False

                self.failure_status_id = first_failure_status_id
            self._report_flaky_tests(tests_that_only_failed_second, second_results_archive)

                return self.report_failure(first_results_archive, first_results, first_script_error)
            if tests_that_consistently_failed:
                if self._should_defer_patch_or_throw(tests_that_consistently_failed, first_results_archive, first_script_error, first_failure_status_id):
                    return False  # Defer patch

            if second_results.did_exceed_test_failure_limit():
                self._should_defer_patch_or_throw(first_results.failing_test_results(), first_results_archive, first_script_error, first_failure_status_id)
                return False

            if first_results.did_exceed_test_failure_limit():
                self._should_defer_patch_or_throw(second_results.failing_test_results(), second_results_archive, second_script_error, second_failure_status_id)
                return False

            if self._results_failed_different_tests(first_results, second_results):
                first_failing_results_set = frozenset(first_results.failing_test_results())
                second_failing_results_set = frozenset(second_results.failing_test_results())

                tests_that_only_failed_first = first_failing_results_set.difference(second_failing_results_set)
                self._report_flaky_tests(tests_that_only_failed_first, first_results_archive)

                tests_that_only_failed_second = second_failing_results_set.difference(first_failing_results_set)
                self._report_flaky_tests(tests_that_only_failed_second, second_results_archive)

                tests_that_consistently_failed = first_failing_results_set.intersection(second_failing_results_set)
                if tests_that_consistently_failed:
                    if self._should_defer_patch_or_throw(tests_that_consistently_failed, first_results_archive, first_script_error, first_failure_status_id):
                        return False  # Defer patch

                # At this point we know that at least one test flaked, but no consistent failures
                # were introduced. This is a bit of a grey-zone.
                return False  # Defer patch

            if self._should_defer_patch_or_throw(first_results.failing_test_results(), first_results_archive, first_script_error, first_failure_status_id):
                return False  # Defer patch

            # At this point, we know that the first and second runs had the exact same failures,
            # and that those failures are all present on the clean tree, so we can say with certainty
            # that the patch is good.
            return True

    def results_archive_from_patch_test_run(self, patch):
        assert(self._patch.id() == patch.id())  # PatchAnalysisTask is not currently re-useable.

Lines 1-4 a/Tools/Scripts/webkitpy/tool/commands/earlywarningsystem.py_sec1
- a/Tools/Scripts/webkitpy/tool/commands/earlywarningsystem.py -3 / +15 lines
1	# Copyright (c) 2009 Google Inc. All rights reserved.	1	# Copyright (c) 2009 Google Inc. All rights reserved.
		2	# Copyright (c) 2017 Apple Inc. All rights reserved.
2	#	3	#
3	# Redistribution and use in source and binary forms, with or without	4	# Redistribution and use in source and binary forms, with or without
4	# modification, are permitted provided that the following conditions are	5	# modification, are permitted provided that the following conditions are
Lines 28-33 a/Tools/Scripts/webkitpy/tool/commands/earlywarningsystem.py_sec2
28		29
29	import json	30	import json
30	import logging	31	import logging
		32	import os
31	from optparse import make_option	33	from optparse import make_option
32		34
33	from webkitpy.common.config.committers import CommitterList	35	from webkitpy.common.config.committers import CommitterList
Lines 36-41 from webkitpy.common.system.filesystem import FileSystem a/Tools/Scripts/webkitpy/tool/commands/earlywarningsystem.py_sec3
36	from webkitpy.common.system.executive import ScriptError	38	from webkitpy.common.system.executive import ScriptError
37	from webkitpy.tool.bot.earlywarningsystemtask import EarlyWarningSystemTask, EarlyWarningSystemTaskDelegate	39	from webkitpy.tool.bot.earlywarningsystemtask import EarlyWarningSystemTask, EarlyWarningSystemTaskDelegate
38	from webkitpy.tool.bot.layouttestresultsreader import LayoutTestResultsReader	40	from webkitpy.tool.bot.layouttestresultsreader import LayoutTestResultsReader
		41	from webkitpy.tool.bot.jsctestresultsreader import JSCTestResultsReader
39	from webkitpy.tool.bot.patchanalysistask import UnableToApplyPatch, PatchIsNotValid	42	from webkitpy.tool.bot.patchanalysistask import UnableToApplyPatch, PatchIsNotValid
40	from webkitpy.tool.bot.queueengine import QueueEngine	43	from webkitpy.tool.bot.queueengine import QueueEngine
41	from webkitpy.tool.commands.queues import AbstractReviewQueue	44	from webkitpy.tool.commands.queues import AbstractReviewQueue
Lines 53-59 class AbstractEarlyWarningSystem(AbstractReviewQueue, EarlyWarningSystemTaskDele a/Tools/Scripts/webkitpy/tool/commands/earlywarningsystem.py_sec4
53		56
54	def begin_work_queue(self):	57	def begin_work_queue(self):
55	AbstractReviewQueue.begin_work_queue(self)	58	AbstractReviewQueue.begin_work_queue(self)
56	self._layout_test_results_reader = LayoutTestResultsReader(self._tool, self._port.results_directory(), self._log_directory())	59
		60	if self.group() == "jsc":
		61	dir = os.path.dirname(os.path.dirname(os.path.dirname(self._port.results_directory())))
		62	self._test_results_reader = JSCTestResultsReader(self._tool, dir)
		63	else:
		64	self._test_results_reader = LayoutTestResultsReader(self._tool, self._port.results_directory(), self._log_directory())
57		65
58	def _failing_tests_message(self, task, patch):	66	def _failing_tests_message(self, task, patch):
59	results = task.results_from_patch_test_run(patch)	67	results = task.results_from_patch_test_run(patch)
Lines 117-130 class AbstractEarlyWarningSystem(AbstractReviewQueue, EarlyWarningSystemTaskDele a/Tools/Scripts/webkitpy/tool/commands/earlywarningsystem.py_sec5
117	return self._update_status(message, patch=patch, results_file=failure_log)	125	return self._update_status(message, patch=patch, results_file=failure_log)
118		126
119	def test_results(self):	127	def test_results(self):
120	return self._layout_test_results_reader.results()	128	return self._test_results_reader.results()
121		129
122	def archive_last_test_results(self, patch):	130	def archive_last_test_results(self, patch):
123	return self._layout_test_results_reader.archive(patch)	131	return self._test_results_reader.archive(patch)
124		132
125	def build_style(self):	133	def build_style(self):
126	return self._build_style	134	return self._build_style
127		135
		136	def group(self):
		137	return self._group
		138
128	def refetch_patch(self, patch):	139	def refetch_patch(self, patch):
129	return self._tool.bugs.fetch_attachment(patch.id())	140	return self._tool.bugs.fetch_attachment(patch.id())
130		141
Lines 156-160 class AbstractEarlyWarningSystem(AbstractReviewQueue, EarlyWarningSystemTaskDele a/Tools/Scripts/webkitpy/tool/commands/earlywarningsystem.py_sec6
156	'_build_style': config.get('style', "release"),	167	'_build_style': config.get('style', "release"),
157	'watchers': config.get('watchers', []),	168	'watchers': config.get('watchers', []),
158	'run_tests': config.get('runTests', cls.run_tests),	169	'run_tests': config.get('runTests', cls.run_tests),
		170	'_group': config.get('group', None),
159	}))	171	}))
160	return classes	172	return classes



# Copyright (C) 2009 Google Inc. All rights reserved.
# Copyright (C) 2017 Apple Inc. All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are

        class TestEWS(AbstractEarlyWarningSystem):
            port_name = "win"  # Needs to be a port which port/factory understands.
            _build_style = None
            _group = None

        ews = TestEWS()
        ews.bind_to_tool(MockTool())

            "port": ews.port_name,
            "architecture": " --architecture=%s" % ews.architecture if ews.architecture else "",
            "build_style": ews.build_style(),
            "group": ews.group(),
        }
        if ews.run_tests:
            run_tests_line = "Running: webkit-patch --status-host=example.com build-and-test --no-clean --no-update --test --non-interactive --build-style=%(build_style)s --group=%(group)s --port=%(port)s%(architecture)s\n" % string_replacements
        else:
            run_tests_line = ""
        string_replacements['run_tests_line'] = run_tests_line

Running: webkit-patch --status-host=example.com clean --port=%(port)s%(architecture)s
Running: webkit-patch --status-host=example.com update --port=%(port)s%(architecture)s
Running: webkit-patch --status-host=example.com apply-attachment --no-update --non-interactive 10000 --port=%(port)s%(architecture)s
Running: webkit-patch --status-host=example.com build --no-clean --no-update --build-style=%(build_style)s --group=%(group)s --port=%(port)s%(architecture)s
%(run_tests_line)sMOCK: update_status: %(name)s Pass
MOCK: release_work_item: %(name)s 10000
""" % string_replacements,



# Copyright (C) 2010 Google Inc. All rights reserved.
# Copyright (C) 2017 Apple Inc. All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are

            Options.build,
            Options.quiet,
            Options.build_style,
            Options.group,
        ]

    def build(self, build_style, group):
        environment = self._tool.copy_current_environment()
        environment.disable_gcc_smartquotes()
        environment.disable_jhbuild_VT100_output()
        env = environment.to_dictionary()

        if group == "jsc":
            build_command = self._tool.deprecated_port().build_jsc_command(build_style=build_style)
        else:
            build_command = self._tool.deprecated_port().build_webkit_command(build_style=build_style)

        if self._options.architecture:
            build_command += ['ARCHS=%s' % self._options.architecture]

        self._tool.executive.run_and_throw_if_fail(build_command, self._options.quiet,
            cwd=self._tool.scm().checkout_root, env=env)

    def run(self, state):
        if not self._options.build:
            return
        _log.info("Building WebKit")

        group = None
        if hasattr(self._options, 'group'):
            group = self._options.group

        if self._options.build_style == "both":
            self.build("debug", group)
            self.build("release", group)
        else:
            self.build(self._options.build_style, group)



# Copyright (C) 2010 Google Inc. All rights reserved.
# Copyright (C) 2017 Apple Inc. All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are

    email = make_option("--email", action="store", type="string", dest="email", help="Email address to use in ChangeLogs.")
    force_clean = make_option("--force-clean", action="store_true", dest="force_clean", default=False, help="Clean working directory before applying patches (removes local changes and commits)")
    git_commit = make_option("-g", "--git-commit", action="store", dest="git_commit", help="Operate on a local commit. If a range, the commits are squashed into one. <ref>.... includes the working copy changes. UPSTREAM can be used for the upstream/tracking branch.")
    group = make_option("--group", action="store", dest="group", default=None, help="")
    local_commit = make_option("--local-commit", action="store_true", dest="local_commit", default=False, help="Make a local commit for each applied patch")
    non_interactive = make_option("--non-interactive", action="store_true", dest="non_interactive", default=False, help="Never prompt the user, fail as fast as possible.")
    obsolete_patches = make_option("--no-obsolete", action="store_false", dest="obsolete_patches", default=True, help="Do not obsolete old patches before posting this one.")

Lines 1-4 a/Tools/Scripts/webkitpy/tool/steps/runtests.py_sec1
- a/Tools/Scripts/webkitpy/tool/steps/runtests.py +10 lines
1	# Copyright (C) 2010 Google Inc. All rights reserved.	1	# Copyright (C) 2010 Google Inc. All rights reserved.
		2	# Copyright (C) 2017 Apple Inc. All rights reserved.
2	#	3	#
3	# Redistribution and use in source and binary forms, with or without	4	# Redistribution and use in source and binary forms, with or without
4	# modification, are permitted provided that the following conditions are	5	# modification, are permitted provided that the following conditions are
Lines 53-58 class RunTests(AbstractStep): a/Tools/Scripts/webkitpy/tool/steps/runtests.py_sec2
53	Options.iterate_on_new_tests,	54	Options.iterate_on_new_tests,
54	Options.non_interactive,	55	Options.non_interactive,
55	Options.quiet,	56	Options.quiet,
		57	Options.group,
56	]	58	]
57		59
58	def run(self, state):	60	def run(self, state):
Lines 63-68 class RunTests(AbstractStep): a/Tools/Scripts/webkitpy/tool/steps/runtests.py_sec3
63	if not self._options.test:	65	if not self._options.test:
64	return	66	return
65		67
		68	if hasattr(self._options, "group") and self._options.group == "jsc":
		69	self._run_javascriptcore_tests()
		70	return
		71
66	python_unittests_command = self._tool.deprecated_port().run_python_unittests_command()	72	python_unittests_command = self._tool.deprecated_port().run_python_unittests_command()
67	if python_unittests_command:	73	if python_unittests_command:
68	_log.info("Running Python unit tests")	74	_log.info("Running Python unit tests")
Lines 143-145 class RunTests(AbstractStep): a/Tools/Scripts/webkitpy/tool/steps/runtests.py_sec4
143	args.append("--iterations=%d" % iterate_on_new_tests)	149	args.append("--iterations=%d" % iterate_on_new_tests)
144		150
145	self._tool.executive.run_and_throw_if_fail(args, cwd=self._tool.scm().checkout_root)	151	self._tool.executive.run_and_throw_if_fail(args, cwd=self._tool.scm().checkout_root)
		152
		153	def _run_javascriptcore_tests(self):
		154	args = self._tool.deprecated_port().run_javascriptcore_tests_command(self._options.build_style)
		155	self._tool.executive.run_and_throw_if_fail(args, cwd=self._tool.scm().checkout_root)



# Copyright (C) 2010 Google Inc. All rights reserved.
# Copyright (C) 2017 Apple Inc. All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are

        self._assert_step_output_with_bug(steps.EnsureBugIsOpenAndAssigned, 50002, expected_logs)

    def test_runtests_args(self):
        self.maxDiff = None
        mock_options = self._step_options()
        mock_options.non_interactive = False
        mock_options.build_style = "release"

Running Perl unit tests
MOCK run_and_throw_if_fail: ['Tools/Scripts/test-webkitperl'], cwd=/mock-checkout
Running JavaScriptCore tests
MOCK run_and_throw_if_fail: ['Tools/Scripts/run-javascriptcore-tests', '--no-fail-fast', '--json-output=jsc_test_results.json'], cwd=/mock-checkout
Running bindings generation tests
MOCK run_and_throw_if_fail: ['Tools/Scripts/run-bindings-tests'], cwd=/mock-checkout
Running run-webkit-tests

        OutputCapture().assert_outputs(self, step.run, [{}], expected_logs=expected_logs)

    def test_runtests_debug_args(self):
        self.maxDiff = None
        mock_options = self._step_options()
        mock_options.non_interactive = False
        mock_options.build_style = "debug"

Running Perl unit tests
MOCK run_and_throw_if_fail: ['Tools/Scripts/test-webkitperl'], cwd=/mock-checkout
Running JavaScriptCore tests
MOCK run_and_throw_if_fail: ['Tools/Scripts/run-javascriptcore-tests', '--no-fail-fast', '--json-output=jsc_test_results.json'], cwd=/mock-checkout
Running bindings generation tests
MOCK run_and_throw_if_fail: ['Tools/Scripts/run-bindings-tests'], cwd=/mock-checkout
Running run-webkit-tests

Return to Bug 162458

Lines 1-3 a/Tools/ChangeLog_sec1
- a/Tools/ChangeLog +82 lines
		1	2017-01-13 Srinivasan Vijayaraghavan <svijayaraghavan@apple.com>
		2
		3	EWS should run JavaScriptCore tests
		4	https://bugs.webkit.org/show_bug.cgi?id=162458
		5
		6	Reviewed by NOBODY (OOPS!).
		7
		8	* Scripts/webkitpy/common/config/ews.json: Add config info for JSC EWS.
		9	* Scripts/webkitpy/common/config/ports.py: Add support for JSC EWS in Mac Port.
		10	(DeprecatedPort.build_jsc_command): Added - command to build JSC only (quicker than building all of WebKit).
		11	(DeprecatedPort.run_javascriptcore_tests_command): Allow JSC EWS to only run JSC tests.
		12	(MacPort.run_webkit_tests_command): Check for JSC.
		13	* Scripts/webkitpy/common/net/jsctestresults.py: Added.
		14	(JSCTestResults):
		15	(JSCTestResults.results_from_string): Use json library to parse results.
		16	(JSCTestResults.__init__): Initialize members.
		17	(JSCTestResults.equals): This enables unit testing.
		18	(JSCTestResults.is_subset): Checks if one set of failures is a subset of another.
		19	(JSCTestResults.stress_test_failures): Getter.
		20	(JSCTestResults.all_api_tests_passed): Getter.
		21	(JSCTestResults.all_passed): Getter.
		22	(JSCTestResults.failing_test_results): Getter. Some of these exist for interopability reasons.
		23	(JSCTestResults.test_results): Getter.
		24	(JSCTestResults.failing_tests): Getter.
		25	(JSCTestResults.did_exceed_test_failure_limit): Getter.
		26	* Scripts/webkitpy/common/net/jsctestresults_unittest.py: Added.
		27	(JSCTestResultsTest): Class to test JSCTestResults.
		28	(JSCTestResultsTest.test_results_from_string): Tests parsing.
		29	* Scripts/webkitpy/tool/bot/jscews_unittest.py: Added.
		30	(MockPatchAnalysisTask): Mocked-out version of PatchAnalysisTask that doesn't run commands.
		31	(MockPatchAnalysisTask.__init__): Sets attributes.
		32	(MockPatchAnalysisTask._test): Override to not run command.
		33	(MockPatchAnalysisTask._build_and_test_without_patch): Override to not run command.
		34	(MockPatchAnalysisTask.validate): Assume mocked patch is valid for purposes of testing retry logic.
		35	(MockPatchAnalysisTask.test_run_count): Specific to the mocked version, to test retry logic.
		36	(MockJSCEarlyWarningSystem): Mocked-out version of AbstarctEarlyWarningSystem so we can provide test results.
		37	(MockJSCEarlyWarningSystem.__init__): Sets attributes, also sets group to jsc.
		38	(MockJSCEarlyWarningSystem.test_results): Returns test results provided by us, instead of using a JSON reader.
		39	(JSCEarlyWarningSystemTest): Class to test retry logic in below situations.
		40	(JSCEarlyWarningSystemTest._create_task): Helper function to abstract out common code.
		41	(JSCEarlyWarningSystemTest.test_success_case): Clean patch on clean tree.
		42	(JSCEarlyWarningSystemTest.test_test_failure): Failed patch on clean tree.
		43	(JSCEarlyWarningSystemTest.test_fix): Patch that fixes all tree redness.
		44	(JSCEarlyWarningSystemTest.test_ineffective_patch): Patch that has same failures as tree.
		45	(JSCEarlyWarningSystemTest.test_partially_effective_patch): Patch fixes some failures but adds no new failures.
		46	(JSCEarlyWarningSystemTest.test_different_test_failures): Patch that has some failures not present in tree.
		47	(JSCEarlyWarningSystemTest.test_first_results_could_not_be_read): Results json file from patch could not be read.
		48	(JSCEarlyWarningSystemTest.test_clean_results_could_not_be_read): Results json file from clean tree could not be read.
		49	* Scripts/webkitpy/tool/bot/jsctestresultsreader.py: Added.
		50	(JSCTestResultsReader): Reads results file.
		51	(JSCTestResultsReader.__init__): Sets attributes.
		52	(JSCTestResultsReader._read_file_contents): Reads file.
		53	(JSCTestResultsReader._create_jsc_test_results): Creates a JSCTestResults object.
		54	(JSCTestResultsReader.results): Invokes _create_jsc_test_results to return a JSCTestResults object.
		55	* Scripts/webkitpy/tool/bot/patchanalysistask.py:
		56	(PatchAnalysisTask._build): Check for JSC.
		57	(PatchAnalysisTask._build_without_patch): Check for JSC.
		58	(PatchAnalysisTask._test): Check for JSC.
		59	(PatchAnalysisTask._build_and_test_without_patch): Check for JSC.
		60	(PatchAnalysisTask._test_patch): Add retry logic for JSC.
		61	* Scripts/webkitpy/tool/commands/earlywarningsystem.py:
		62	(AbstractEarlyWarningSystem.begin_work_queue): Use JSCTestResultsReader instead of LayoutTestResultsReader in JSC EWS.
		63	(AbstractEarlyWarningSystem.test_results): _layout_test_results_reader -> _test_results_reader.
		64	(AbstractEarlyWarningSystem.archive_last_test_results): _layout_test_results_reader -> _test_results_reader.
65	(AbstractEarlyWarningSystem.group): This attribute determines the type of EWS (eg. JSC).
66	(AbstractEarlyWarningSystem.load_ews_classes): Add _group.
67	* Scripts/webkitpy/tool/commands/earlywarningsystem_unittest.py:
68	(EarlyWarningSystemTest._default_expected_logs): Add _group.
69	* Scripts/webkitpy/tool/steps/build.py:
70	(Build.options): Check for JSC.
71	(Build.build): Check for JSC.
72	(Build.run): Check for JSC.
73	* Scripts/webkitpy/tool/steps/options.py:
74	(Options): Add --group command line option.
75	* Scripts/webkitpy/tool/steps/runtests.py:
76	(RunTests.options): Add group.
77	(RunTests.run): Check for JSC.
78	(RunTests._run_webkit_tests): Check for JSC.
79	(RunTests): Add _group attribute.
80	(RunTests._run_javascriptcore_tests): New.
81	* Scripts/webkitpy/tool/steps/steps_unittest.py: Change expected command for running jsc tests.
82
1	2016-09-25 Gyuyoung Kim <gyuyoung.kim@webkit.org>	83	2016-09-25 Gyuyoung Kim <gyuyoung.kim@webkit.org>
2		84
3	[EFL] Adjust EFL indentation to EFL MiniBrowser	85	[EFL] Adjust EFL indentation to EFL MiniBrowser

Line 0 a/Tools/Scripts/webkitpy/common/net/jsctestresults.py_sec1
- a/Tools/Scripts/webkitpy/common/net/jsctestresults.py +76 lines
		1	# Copyright (C) 2017 Apple Inc. All rights reserved.
		2	#
		3	# Redistribution and use in source and binary forms, with or without
		4	# modification, are permitted provided that the following conditions
		5	# are met:
		6	# 1. Redistributions of source code must retain the above copyright
		7	# notice, this list of conditions and the following disclaimer.
		8	# 2. Redistributions in binary form must reproduce the above copyright
		9	# notice, this list of conditions and the following disclaimer in the
		10	# documentation and/or other materials provided with the distribution.
		11	#
		12	# THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS'' AND
		13	# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
		14	# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
		15	# DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS BE LIABLE FOR
		16	# ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
		17	# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
		18	# SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
		19	# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
		20	# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
		21	# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
		22
		23	import json
		24	import logging
		25
		26	_log = logging.getLogger(__name__)
		27
		28
		29	class JSCTestResults(object):
		30	@classmethod
		31	def results_from_string(cls, string):
		32	if not string:
		33	return None
		34
		35	try:
		36	parsed_results = json.loads(string)
		37	except ValueError:
		38	return None
		39
		40	return cls(parsed_results['allApiTestsPassed'], parsed_results['stressTestFailures'])
		41
		42	def __init__(self, all_api_tests_passed, stress_test_failures):
		43	self._all_api_tests_passed = all_api_tests_passed
		44	self._stress_test_failures = stress_test_failures
		45
		46	self._test_results = stress_test_failures[:]
		47	if not self._all_api_tests_passed:
		48	self._test_results.append("apiTests")
		49
		50	def equals(self, other):
		51	return self._all_api_tests_passed == other._all_api_tests_passed and set(self._stress_test_failures) == set(other._stress_test_failures)
		52
		53	def is_subset(self, other):
		54	return set(self.test_results()) <= set(other.test_results())
		55
		56	def stress_test_failures(self):
		57	return self._stress_test_failures
		58
		59	def all_api_tests_passed(self):
		60	return self._all_api_tests_passed
		61
		62	def all_passed(self):
		63	return self._all_api_tests_passed and not self._stress_test_failures
		64
65	def failing_test_results(self):
66	return self.stress_test_failures()
67
68	def test_results(self):
69	return self._test_results
70
71	def failing_tests(self):
72	return self._test_results
73
74	# No defined failure limit for JSC.
75	def did_exceed_test_failure_limit(self):
76	return False

Line 0 a/Tools/Scripts/webkitpy/tool/bot/jscews_unittest.py_sec1
- a/Tools/Scripts/webkitpy/tool/bot/jscews_unittest.py +155 lines
		1	# Copyright (C) 2017 Apple Inc. All rights reserved.
		2	#
		3	# Redistribution and use in source and binary forms, with or without
		4	# modification, are permitted provided that the following conditions
		5	# are met:
		6	# 1. Redistributions of source code must retain the above copyright
		7	# notice, this list of conditions and the following disclaimer.
		8	# 2. Redistributions in binary form must reproduce the above copyright
		9	# notice, this list of conditions and the following disclaimer in the
		10	# documentation and/or other materials provided with the distribution.
		11	#
		12	# THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS'' AND
		13	# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
		14	# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
		15	# DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS BE LIABLE FOR
		16	# ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
		17	# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
		18	# SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
		19	# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
		20	# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
		21	# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
		22
		23	import logging
		24	import unittest
		25
		26	from webkitpy.common.net.jsctestresults import JSCTestResults
		27	from webkitpy.common.system.executive import ScriptError
		28	from webkitpy.tool.bot.patchanalysistask import *
		29	from webkitpy.tool.commands.earlywarningsystem import AbstractEarlyWarningSystem
		30	from webkitpy.tool.mocktool import MockTool
		31
		32	_log = logging.getLogger(__name__)
		33
		34
		35	class MockPatchAnalysisTask(PatchAnalysisTask):
		36	def __init__(self, delegate, patch, first_test_all_passed):
		37	self._delegate = delegate
		38	self._patch = patch
		39	self._script_error = None
		40	self._results_archive_from_patch_test_run = None
		41	self._results_from_patch_test_run = None
		42	self.error = None
		43	self._first_test_all_passed = first_test_all_passed
		44	self._test_run_count = 0
		45
		46	def _test(self):
		47	self._test_run_count += 1
		48
		49	if self._first_test_all_passed == True:
		50	return True
		51	else:
		52	self._script_error = ScriptError("Regression test")
		53	return False
		54
		55	def _build_and_test_without_patch(self):
		56	self._test_run_count += 1
		57	return True
		58
		59	def validate(self):
		60	return True
		61
		62	def test_run_count(self):
		63	return self._test_run_count
		64
65
66	# This is the delegate to be used with MockPatchAnalysisTask, above.
67	class MockJSCEarlyWarningSystem(AbstractEarlyWarningSystem):
68	def __init__(self, first_test_results, clean_test_results):
69	AbstractEarlyWarningSystem.__init__(self)
70	self._group = "jsc"
71	self._results_in_order = [clean_test_results, first_test_results]
72
73	def test_results(self):
74	return self._results_in_order.pop()
75
76
77	class JSCEarlyWarningSystemTest(unittest.TestCase):
78	def _create_task(self, first_test_results, clean_test_results):
79	queue = MockJSCEarlyWarningSystem(first_test_results, clean_test_results)
80	tool = MockTool(log_executive=True)
81	patch = tool.bugs.fetch_attachment(10000)
82	patch_passed_all_tests = False if first_test_results is None else first_test_results.all_passed()
83	return MockPatchAnalysisTask(queue, patch, patch_passed_all_tests)
84
85	def test_success_case(self):
86	first_test_results = JSCTestResults(True, [])
87	clean_test_results = JSCTestResults(True, [])
88	task = self._create_task(first_test_results, clean_test_results)
89
90	return_value = task._test_patch()
91	self.assertEqual(task.test_run_count(), 1)
92	self.assertEqual(return_value, True)
93
94	def test_test_failure(self):
95	first_test_results = JSCTestResults(True, ['Fail.js'])
96	clean_test_results = JSCTestResults(True, [])
97	task = self._create_task(first_test_results, clean_test_results)
98
99	with self.assertRaises(ScriptError):
100	return_value = task._test_patch()
101	self.assertEqual(task.test_run_count(), 2)
102
103	def test_fix(self):
104	first_test_results = JSCTestResults(True, [])
105	clean_test_results = JSCTestResults(True, ['Fail.js'])
106	task = self._create_task(first_test_results, clean_test_results)
107
108	return_value = task._test_patch()
109	self.assertEqual(task.test_run_count(), 1)
110	self.assertEqual(return_value, True)
111
112	def test_ineffective_patch(self):
113	first_test_results = JSCTestResults(False, ['failure1.js', 'failure2.js'])
114	clean_test_results = JSCTestResults(False, ['failure1.js', 'failure2.js'])
115	task = self._create_task(first_test_results, clean_test_results)
116
117	return_value = task._test_patch()
118	self.assertEqual(task.test_run_count(), 2)
119	self.assertEqual(return_value, True)
120
121	def test_partially_effective_patch(self):
122	first_test_results = JSCTestResults(True, ['failure2.js'])
123	clean_test_results = JSCTestResults(False, ['failure1.js', 'failure2.js'])
124	task = self._create_task(first_test_results, clean_test_results)
125
126	return_value = task._test_patch()
127	self.assertEqual(task.test_run_count(), 2)
128	self.assertEqual(return_value, True)
129
130	def test_different_test_failures(self):
131	first_test_results = JSCTestResults(False, [])
132	clean_test_results = JSCTestResults(True, ['failure1.js', 'failure2.js'])
133	task = self._create_task(first_test_results, clean_test_results)
134
135	with self.assertRaises(ScriptError):
136	return_value = task._test_patch()
137	self.assertEqual(task.test_run_count(), 2)
138
139	def test_first_results_could_not_be_read(self):
140	first_test_results = None
141	clean_test_results = JSCTestResults(True, [])
142	task = self._create_task(first_test_results, clean_test_results)
143
144	with self.assertRaises(ScriptError):
145	return_value = task._test_patch()
146	self.assertEqual(task.test_run_count(), 1)
147
148	def test_clean_results_could_not_be_read(self):
149	first_test_results = JSCTestResults(True, ['failure2.js'])
150	clean_test_results = None
151	task = self._create_task(first_test_results, clean_test_results)
152
153	with self.assertRaises(ScriptError):
154	return_value = task._test_patch()
155	self.assertEqual(task.test_run_count(), 2)

Line 0 a/Tools/Scripts/webkitpy/tool/bot/jsctestresultsreader.py_sec1
- a/Tools/Scripts/webkitpy/tool/bot/jsctestresultsreader.py +47 lines
	1	# Copyright (C) 2017 Apple Inc. All rights reserved.
	2	#
	3	# Redistribution and use in source and binary forms, with or without
	4	# modification, are permitted provided that the following conditions
	5	# are met:
	6	# 1. Redistributions of source code must retain the above copyright
	7	# notice, this list of conditions and the following disclaimer.
	8	# 2. Redistributions in binary form must reproduce the above copyright
	9	# notice, this list of conditions and the following disclaimer in the
	10	# documentation and/or other materials provided with the distribution.
	11	#
	12	# THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS'' AND
	13	# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
	14	# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
	15	# DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS BE LIABLE FOR
	16	# ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
	17	# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
	18	# SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
	19	# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
	20	# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
	21	# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
	22
	23	import logging
	24
	25	from webkitpy.common.net.jsctestresults import JSCTestResults
	26
	27	_log = logging.getLogger(__name__)
	28
	29
	30	class JSCTestResultsReader(object):
	31	def __init__(self, host, results_directory):
	32	self._host = host
	33	self._results_directory = results_directory
	34
	35	def _read_file_contents(self, path):
	36	try:
	37	return self._host.filesystem.read_text_file(path)
	38	except IOError, e: # File does not exist or can't be read.
	39	return None
	40
	41	def _create_jsc_test_results(self):
	42	results_path = self._host.filesystem.join(self._results_directory, "jsc_test_results.json")
	43	contents = self._read_file_contents(results_path)
	44	return JSCTestResults.results_from_string(contents)
	45
	46	def results(self):
	47	return self._create_jsc_test_results()

Lines 1-4 a/Tools/Scripts/webkitpy/tool/bot/patchanalysistask.py_sec1
- a/Tools/Scripts/webkitpy/tool/bot/patchanalysistask.py -63 / +111 lines
1	# Copyright (c) 2010 Google Inc. All rights reserved.	1	# Copyright (c) 2010 Google Inc. All rights reserved.
		2	# Copyright (C) 2017 Apple Inc. All rights reserved.
2	#	3	#
3	# Redistribution and use in source and binary forms, with or without	4	# Redistribution and use in source and binary forms, with or without
4	# modification, are permitted provided that the following conditions are	5	# modification, are permitted provided that the following conditions are
Lines 123-149 class PatchAnalysisTask(object): a/Tools/Scripts/webkitpy/tool/bot/patchanalysistask.py_sec2
123	"Patch does not apply")	124	"Patch does not apply")
124		125
125	def _build(self):	126	def _build(self):
126	return self._run_command([	127	args = [
127	"build",	128	"build",
128	"--no-clean",	129	"--no-clean",
129	"--no-update",	130	"--no-update",
130	"--build-style=%s" % self._delegate.build_style(),	131	"--build-style=%s" % self._delegate.build_style(),
131	],	132	]
132	"Built patch",	133
133	"Patch does not build")	134	if hasattr(self._delegate, 'group'):
		135	args.append("--group=%s" % self._delegate.group())
		136
		137	return self._run_command(
		138	args,
		139	"Built patch",
		140	"Patch does not build")
134		141
135	def _build_without_patch(self):	142	def _build_without_patch(self):
136	return self._run_command([	143	args = [
137	"build",	144	"build",
138	"--force-clean",	145	"--force-clean",
139	"--no-update",	146	"--no-update",
140	"--build-style=%s" % self._delegate.build_style(),	147	"--build-style=%s" % self._delegate.build_style(),
141	],	148	]
142	"Able to build without patch",	149
143	"Unable to build without patch")	150	if hasattr(self._delegate, 'group'):
		151	args.append("--group=%s" % self._delegate.group())
		152
		153	return self._run_command(
		154	args,
		155	"Able to build without patch",
		156	"Unable to build without patch")
144		157
145	def _test(self):	158	def _test(self):
146	return self._run_command([	159	args = [
147	"build-and-test",	160	"build-and-test",
148	"--no-clean",	161	"--no-clean",
149	"--no-update",	162	"--no-update",
Lines 151-162 class PatchAnalysisTask(object): a/Tools/Scripts/webkitpy/tool/bot/patchanalysistask.py_sec3
151	"--test",	164	"--test",
152	"--non-interactive",	165	"--non-interactive",
153	"--build-style=%s" % self._delegate.build_style(),	166	"--build-style=%s" % self._delegate.build_style(),
154	],	167	]
155	"Passed tests",	168
156	"Patch does not pass tests")	169	if hasattr(self._delegate, 'group'):
		170	args.append("--group=%s" % self._delegate.group())
		171
		172	return self._run_command(
		173	args,
		174	"Passed tests",
		175	"Patch does not pass tests")
157		176
158	def _build_and_test_without_patch(self):	177	def _build_and_test_without_patch(self):
159	return self._run_command([	178	args = [
160	"build-and-test",	179	"build-and-test",
161	"--force-clean",	180	"--force-clean",
162	"--no-update",	181	"--no-update",
Lines 164-172 class PatchAnalysisTask(object): a/Tools/Scripts/webkitpy/tool/bot/patchanalysistask.py_sec4
164	"--test",	183	"--test",
165	"--non-interactive",	184	"--non-interactive",
166	"--build-style=%s" % self._delegate.build_style(),	185	"--build-style=%s" % self._delegate.build_style(),
167	],	186	]
168	"Able to pass tests without patch",	187
169	"Unable to pass tests without patch (tree is red?)")	188	if hasattr(self._delegate, 'group'):
		189	args.append("--group=%s" % self._delegate.group())
		190
		191	return self._run_command(
		192	args,
		193	"Able to pass tests without patch",
		194	"Unable to pass tests without patch (tree is red?)")
170		195
171	def _land(self):	196	def _land(self):
172	# Unclear if this should pass --quiet or not. If --parent-command always does the reporting, then it should.	197	# Unclear if this should pass --quiet or not. If --parent-command always does the reporting, then it should.
Lines 210-278 class PatchAnalysisTask(object): a/Tools/Scripts/webkitpy/tool/bot/patchanalysistask.py_sec5
210	if self._test():	235	if self._test():
211	return True	236	return True
212		237
213	# Note: archive_last_test_results deletes the results directory, making these calls order-sensitve.	238	# FIXME: Abstract out the retry logic instead of casing on the test suite here.
214	# We could remove this dependency by building the test_results from the archive.	239	if hasattr(self._delegate, 'group') and self._delegate.group() == "jsc":
215	first_results = self._delegate.test_results()	240	first_results = self._delegate.test_results()
216	first_results_archive = self._delegate.archive_last_test_results(self._patch)	241	first_script_error = self._script_error
217	first_script_error = self._script_error	242
218	first_failure_status_id = self.failure_status_id	243	if first_results is None:
219		244	return self.report_failure(None, None, ScriptError("Could not read JSC test results for patch."))
220	if self._test() and not first_results.did_exceed_test_failure_limit():
221	# Only report flaky tests if we were successful at parsing results.json and archiving results.
222	if first_results and first_results_archive:
223	self._report_flaky_tests(first_results.failing_test_results(), first_results_archive)
224	return True
225		245
226	second_results = self._delegate.test_results()	246	if first_results.all_passed():
227	second_results_archive = self._delegate.archive_last_test_results(self._patch)	247	return True
228	second_script_error = self._script_error
229	second_failure_status_id = self.failure_status_id
230		248
231	if second_results.did_exceed_test_failure_limit() and first_results.did_exceed_test_failure_limit():
232	self._build_and_test_without_patch()	249	self._build_and_test_without_patch()
233	clean_tree_results = self._delegate.test_results()	250	clean_tree_results = self._delegate.test_results()
234		251
235	if (len(first_results.failing_tests()) - len(clean_tree_results.failing_tests())) <= 5:	252	if clean_tree_results is None:
236	return False	253	return self.report_failure(None, None, ScriptError("Could not read JSC test results for clean tree."))
237		254
238	self.failure_status_id = first_failure_status_id	255	if first_results.is_subset(clean_tree_results):
		256	return True
239		257
240	return self.report_failure(first_results_archive, first_results, first_script_error)	258	return self.report_failure(None, first_results, first_script_error)
241		259
242	if second_results.did_exceed_test_failure_limit():	260	else:
243	self._should_defer_patch_or_throw(first_results.failing_test_results(), first_results_archive, first_script_error, first_failure_status_id)	261	# Note: archive_last_test_results deletes the results directory, making these calls order-sensitve.
244	return False	262	# We could remove this dependency by building the test_results from the archive.
		263	first_results = self._delegate.test_results()
		264	first_results_archive = self._delegate.archive_last_test_results(self._patch)
		265	first_script_error = self._script_error
		266	first_failure_status_id = self.failure_status_id
245		267
246	if first_results.did_exceed_test_failure_limit():	268	if self._test() and not first_results.did_exceed_test_failure_limit():
247	self._should_defer_patch_or_throw(second_results.failing_test_results(), second_results_archive, second_script_error, second_failure_status_id)	269	# Only report flaky tests if we were successful at parsing results.json and archiving results.
248	return False	270	if first_results and first_results_archive:
		271	self._report_flaky_tests(first_results.failing_test_results(), first_results_archive)
		272	return True
		273
		274	second_results = self._delegate.test_results()
		275	second_results_archive = self._delegate.archive_last_test_results(self._patch)
		276	second_script_error = self._script_error
		277	second_failure_status_id = self.failure_status_id
249		278
250	if self._results_failed_different_tests(first_results, second_results):	279	if second_results.did_exceed_test_failure_limit() and first_results.did_exceed_test_failure_limit():
251	first_failing_results_set = frozenset(first_results.failing_test_results())	280	self._build_and_test_without_patch()
252	second_failing_results_set = frozenset(second_results.failing_test_results())	281	clean_tree_results = self._delegate.test_results()
253		282
254	tests_that_only_failed_first = first_failing_results_set.difference(second_failing_results_set)	283	if (len(first_results.failing_tests()) - len(clean_tree_results.failing_tests())) <= 5:
255	self._report_flaky_tests(tests_that_only_failed_first, first_results_archive)	284	return False
256		285
257	tests_that_only_failed_second = second_failing_results_set.difference(first_failing_results_set)	286	self.failure_status_id = first_failure_status_id
258	self._report_flaky_tests(tests_that_only_failed_second, second_results_archive)
259		287
260	tests_that_consistently_failed = first_failing_results_set.intersection(second_failing_results_set)	288	return self.report_failure(first_results_archive, first_results, first_script_error)
261	if tests_that_consistently_failed:
262	if self._should_defer_patch_or_throw(tests_that_consistently_failed, first_results_archive, first_script_error, first_failure_status_id):
263	return False # Defer patch
264		289
265	# At this point we know that at least one test flaked, but no consistent failures	290	if second_results.did_exceed_test_failure_limit():
266	# were introduced. This is a bit of a grey-zone.	291	self._should_defer_patch_or_throw(first_results.failing_test_results(), first_results_archive, first_script_error, first_failure_status_id)
267	return False # Defer patch	292	return False
268		293
269	if self._should_defer_patch_or_throw(first_results.failing_test_results(), first_results_archive, first_script_error, first_failure_status_id):	294	if first_results.did_exceed_test_failure_limit():
270	return False # Defer patch	295	self._should_defer_patch_or_throw(second_results.failing_test_results(), second_results_archive, second_script_error, second_failure_status_id)
		296	return False
271		297
272	# At this point, we know that the first and second runs had the exact same failures,	298	if self._results_failed_different_tests(first_results, second_results):
273	# and that those failures are all present on the clean tree, so we can say with certainty	299	first_failing_results_set = frozenset(first_results.failing_test_results())
274	# that the patch is good.	300	second_failing_results_set = frozenset(second_results.failing_test_results())
275	return True	301
		302	tests_that_only_failed_first = first_failing_results_set.difference(second_failing_results_set)
		303	self._report_flaky_tests(tests_that_only_failed_first, first_results_archive)
		304
		305	tests_that_only_failed_second = second_failing_results_set.difference(first_failing_results_set)
		306	self._report_flaky_tests(tests_that_only_failed_second, second_results_archive)
		307
		308	tests_that_consistently_failed = first_failing_results_set.intersection(second_failing_results_set)
		309	if tests_that_consistently_failed:
		310	if self._should_defer_patch_or_throw(tests_that_consistently_failed, first_results_archive, first_script_error, first_failure_status_id):
		311	return False # Defer patch
		312
		313	# At this point we know that at least one test flaked, but no consistent failures
		314	# were introduced. This is a bit of a grey-zone.
		315	return False # Defer patch
		316
		317	if self._should_defer_patch_or_throw(first_results.failing_test_results(), first_results_archive, first_script_error, first_failure_status_id):
		318	return False # Defer patch
		319
		320	# At this point, we know that the first and second runs had the exact same failures,
		321	# and that those failures are all present on the clean tree, so we can say with certainty
		322	# that the patch is good.
		323	return True
276		324
277	def results_archive_from_patch_test_run(self, patch):	325	def results_archive_from_patch_test_run(self, patch):
278	assert(self._patch.id() == patch.id()) # PatchAnalysisTask is not currently re-useable.	326	assert(self._patch.id() == patch.id()) # PatchAnalysisTask is not currently re-useable.

Lines 1-4 a/Tools/Scripts/webkitpy/tool/steps/build.py_sec1
- a/Tools/Scripts/webkitpy/tool/steps/build.py -7 / +17 lines
1	# Copyright (C) 2010 Google Inc. All rights reserved.	1	# Copyright (C) 2010 Google Inc. All rights reserved.
		2	# Copyright (C) 2017 Apple Inc. All rights reserved.
2	#	3	#
3	# Redistribution and use in source and binary forms, with or without	4	# Redistribution and use in source and binary forms, with or without
4	# modification, are permitted provided that the following conditions are	5	# modification, are permitted provided that the following conditions are
Lines 41-68 class Build(AbstractStep): a/Tools/Scripts/webkitpy/tool/steps/build.py_sec2
41	Options.build,	42	Options.build,
42	Options.quiet,	43	Options.quiet,
43	Options.build_style,	44	Options.build_style,
		45	Options.group,
44	]	46	]
45		47
46	def build(self, build_style):	48	def build(self, build_style, group):
47	environment = self._tool.copy_current_environment()	49	environment = self._tool.copy_current_environment()
48	environment.disable_gcc_smartquotes()	50	environment.disable_gcc_smartquotes()
49	environment.disable_jhbuild_VT100_output()	51	environment.disable_jhbuild_VT100_output()
50	env = environment.to_dictionary()	52	env = environment.to_dictionary()
51		53
52	build_webkit_command = self._tool.deprecated_port().build_webkit_command(build_style=build_style)	54	if group == "jsc":
		55	build_command = self._tool.deprecated_port().build_jsc_command(build_style=build_style)
		56	else:
		57	build_command = self._tool.deprecated_port().build_webkit_command(build_style=build_style)
53		58
54	if self._options.architecture:	59	if self._options.architecture:
55	build_webkit_command += ['ARCHS=%s' % self._options.architecture]	60	build_command += ['ARCHS=%s' % self._options.architecture]
56		61
57	self._tool.executive.run_and_throw_if_fail(build_webkit_command, self._options.quiet,	62	self._tool.executive.run_and_throw_if_fail(build_command, self._options.quiet,
58	cwd=self._tool.scm().checkout_root, env=env)	63	cwd=self._tool.scm().checkout_root, env=env)
59		64
60	def run(self, state):	65	def run(self, state):
61	if not self._options.build:	66	if not self._options.build:
62	return	67	return
63	_log.info("Building WebKit")	68	_log.info("Building WebKit")
		69
		70	group = None
		71	if hasattr(self._options, 'group'):
		72	group = self._options.group
		73
64	if self._options.build_style == "both":	74	if self._options.build_style == "both":
65	self.build("debug")	75	self.build("debug", group)
66	self.build("release")	76	self.build("release", group)
67	else:	77	else:
68	self.build(self._options.build_style)	78	self.build(self._options.build_style, group)

Lines 1-4 a/Tools/Scripts/webkitpy/tool/steps/options.py_sec1
- a/Tools/Scripts/webkitpy/tool/steps/options.py +2 lines
1	# Copyright (C) 2010 Google Inc. All rights reserved.	1	# Copyright (C) 2010 Google Inc. All rights reserved.
		2	# Copyright (C) 2017 Apple Inc. All rights reserved.
2	#	3	#
3	# Redistribution and use in source and binary forms, with or without	4	# Redistribution and use in source and binary forms, with or without
4	# modification, are permitted provided that the following conditions are	5	# modification, are permitted provided that the following conditions are
Lines 46-51 class Options(object): a/Tools/Scripts/webkitpy/tool/steps/options.py_sec2
46	email = make_option("--email", action="store", type="string", dest="email", help="Email address to use in ChangeLogs.")	47	email = make_option("--email", action="store", type="string", dest="email", help="Email address to use in ChangeLogs.")
47	force_clean = make_option("--force-clean", action="store_true", dest="force_clean", default=False, help="Clean working directory before applying patches (removes local changes and commits)")	48	force_clean = make_option("--force-clean", action="store_true", dest="force_clean", default=False, help="Clean working directory before applying patches (removes local changes and commits)")
48	git_commit = make_option("-g", "--git-commit", action="store", dest="git_commit", help="Operate on a local commit. If a range, the commits are squashed into one. <ref>.... includes the working copy changes. UPSTREAM can be used for the upstream/tracking branch.")	49	git_commit = make_option("-g", "--git-commit", action="store", dest="git_commit", help="Operate on a local commit. If a range, the commits are squashed into one. <ref>.... includes the working copy changes. UPSTREAM can be used for the upstream/tracking branch.")
		50	group = make_option("--group", action="store", dest="group", default=None, help="")
49	local_commit = make_option("--local-commit", action="store_true", dest="local_commit", default=False, help="Make a local commit for each applied patch")	51	local_commit = make_option("--local-commit", action="store_true", dest="local_commit", default=False, help="Make a local commit for each applied patch")
50	non_interactive = make_option("--non-interactive", action="store_true", dest="non_interactive", default=False, help="Never prompt the user, fail as fast as possible.")	52	non_interactive = make_option("--non-interactive", action="store_true", dest="non_interactive", default=False, help="Never prompt the user, fail as fast as possible.")
51	obsolete_patches = make_option("--no-obsolete", action="store_false", dest="obsolete_patches", default=True, help="Do not obsolete old patches before posting this one.")	53	obsolete_patches = make_option("--no-obsolete", action="store_false", dest="obsolete_patches", default=True, help="Do not obsolete old patches before posting this one.")

Lines 1-4 a/Tools/Scripts/webkitpy/tool/steps/steps_unittest.py_sec1
- a/Tools/Scripts/webkitpy/tool/steps/steps_unittest.py -2 / +5 lines
1	# Copyright (C) 2010 Google Inc. All rights reserved.	1	# Copyright (C) 2010 Google Inc. All rights reserved.
		2	# Copyright (C) 2017 Apple Inc. All rights reserved.
2	#	3	#
3	# Redistribution and use in source and binary forms, with or without	4	# Redistribution and use in source and binary forms, with or without
4	# modification, are permitted provided that the following conditions are	5	# modification, are permitted provided that the following conditions are
Lines 98-103 class StepsTest(unittest.TestCase): a/Tools/Scripts/webkitpy/tool/steps/steps_unittest.py_sec2
98	self._assert_step_output_with_bug(steps.EnsureBugIsOpenAndAssigned, 50002, expected_logs)	99	self._assert_step_output_with_bug(steps.EnsureBugIsOpenAndAssigned, 50002, expected_logs)
99		100
100	def test_runtests_args(self):	101	def test_runtests_args(self):
		102	self.maxDiff = None
101	mock_options = self._step_options()	103	mock_options = self._step_options()
102	mock_options.non_interactive = False	104	mock_options.non_interactive = False
103	mock_options.build_style = "release"	105	mock_options.build_style = "release"
Lines 111-117 MOCK run_and_throw_if_fail: ['Tools/Scripts/test-webkitpy'], cwd=/mock-checkout a/Tools/Scripts/webkitpy/tool/steps/steps_unittest.py_sec3
111	Running Perl unit tests	113	Running Perl unit tests
112	MOCK run_and_throw_if_fail: ['Tools/Scripts/test-webkitperl'], cwd=/mock-checkout	114	MOCK run_and_throw_if_fail: ['Tools/Scripts/test-webkitperl'], cwd=/mock-checkout
113	Running JavaScriptCore tests	115	Running JavaScriptCore tests
114	MOCK run_and_throw_if_fail: ['Tools/Scripts/run-javascriptcore-tests'], cwd=/mock-checkout	116	MOCK run_and_throw_if_fail: ['Tools/Scripts/run-javascriptcore-tests', '--no-fail-fast', '--json-output=jsc_test_results.json'], cwd=/mock-checkout
115	Running bindings generation tests	117	Running bindings generation tests
116	MOCK run_and_throw_if_fail: ['Tools/Scripts/run-bindings-tests'], cwd=/mock-checkout	118	MOCK run_and_throw_if_fail: ['Tools/Scripts/run-bindings-tests'], cwd=/mock-checkout
117	Running run-webkit-tests	119	Running run-webkit-tests
Lines 120-125 MOCK run_and_throw_if_fail: ['Tools/Scripts/run-webkit-tests', '--release', '--q a/Tools/Scripts/webkitpy/tool/steps/steps_unittest.py_sec4
120	OutputCapture().assert_outputs(self, step.run, [{}], expected_logs=expected_logs)	122	OutputCapture().assert_outputs(self, step.run, [{}], expected_logs=expected_logs)
121		123
122	def test_runtests_debug_args(self):	124	def test_runtests_debug_args(self):
		125	self.maxDiff = None
123	mock_options = self._step_options()	126	mock_options = self._step_options()
124	mock_options.non_interactive = False	127	mock_options.non_interactive = False
125	mock_options.build_style = "debug"	128	mock_options.build_style = "debug"
Lines 133-139 MOCK run_and_throw_if_fail: ['Tools/Scripts/test-webkitpy'], cwd=/mock-checkout a/Tools/Scripts/webkitpy/tool/steps/steps_unittest.py_sec5
133	Running Perl unit tests	136	Running Perl unit tests
134	MOCK run_and_throw_if_fail: ['Tools/Scripts/test-webkitperl'], cwd=/mock-checkout	137	MOCK run_and_throw_if_fail: ['Tools/Scripts/test-webkitperl'], cwd=/mock-checkout
135	Running JavaScriptCore tests	138	Running JavaScriptCore tests
136	MOCK run_and_throw_if_fail: ['Tools/Scripts/run-javascriptcore-tests'], cwd=/mock-checkout	139	MOCK run_and_throw_if_fail: ['Tools/Scripts/run-javascriptcore-tests', '--no-fail-fast', '--json-output=jsc_test_results.json'], cwd=/mock-checkout
137	Running bindings generation tests	140	Running bindings generation tests
138	MOCK run_and_throw_if_fail: ['Tools/Scripts/run-bindings-tests'], cwd=/mock-checkout	141	MOCK run_and_throw_if_fail: ['Tools/Scripts/run-bindings-tests'], cwd=/mock-checkout
139	Running run-webkit-tests	142	Running run-webkit-tests