Attachment #298739 for bug #162458

View Details Raw Unified Return to Bug 162458

| Differences between



2017-01-12  Srinivasan Vijayaraghavan  <svijayaraghavan@apple.com>

        EWS should run JavaScriptCore tests
        https://bugs.webkit.org/show_bug.cgi?id=162458

        Reviewed by NOBODY (OOPS!).

        * Scripts/webkitpy/common/config/ews.json: Add config info for JSC EWS.
        * Scripts/webkitpy/common/config/ports.py: Add support for JSC EWS in Mac Port.
        (DeprecatedPort.build_jsc_command): Added - command to build JSC only (quicker than building all of WebKit).
        (DeprecatedPort.run_javascriptcore_tests_command): Allow JSC EWS to only run JSC tests.
        (MacPort.run_webkit_tests_command): Check for JSC.
        * Scripts/webkitpy/common/net/jsctestresults.py: Added.
        (JSCTestResults):
        (JSCTestResults.results_from_string): Use json library to parse results.
        (JSCTestResults.__init__): Initialize members.
        (JSCTestResults.__eq__): This enables unit testing.
        (JSCTestResults.stress_test_failures): Getter.
        (JSCTestResults.all_api_tests_passed): Getter.
        (JSCTestResults.all_passed): Getter.
        (JSCTestResults.failing_test_results): Getter. Some of these exist for interopability reasons.
        (JSCTestResults.test_results): Getter.
        (JSCTestResults.failing_tests): Getter.
        (JSCTestResults.did_exceed_test_failure_limit): Getter.
        * Scripts/webkitpy/tool/bot/jscews_unittest.py: Added.
        (MockPatchAnalysisTask): Mocked-out version of PatchAnalysisTask that doesn't run commands.
        (MockPatchAnalysisTask.__init__): Sets attributes.
        (MockPatchAnalysisTask._test): Override to not run command.
        (MockPatchAnalysisTask._build_and_test_without_patch): Override to not run command.
        (MockPatchAnalysisTask.validate): Assume mocked patch is valid for purposes of testing retry logic.
        (MockPatchAnalysisTask.test_run_count): Specific to the mocked version, to test retry logic.
        (MockJSCEarlyWarningSystem): Mocked-out version of AbstarctEarlyWarningSystem so we can provide test results.
        (MockJSCEarlyWarningSystem.__init__): Sets attributes, also sets group to jsc.
        (MockJSCEarlyWarningSystem.test_results): Returns test results provided by us, instead of using a JSON reader.
        (JSCEarlyWarningSystemTest): Class to test retry logic in below situations.
        (JSCEarlyWarningSystemTest._create_task): Helper function to abstract out common code.
        (JSCEarlyWarningSystemTest.test_success_case): Clean patch on clean tree.
        (JSCEarlyWarningSystemTest.test_test_failure): Failed patch on clean tree.
        (JSCEarlyWarningSystemTest.test_fix): Patch that fixes all tree redness.
        (JSCEarlyWarningSystemTest.test_ineffective_patch): Patch that has same failures as tree.
        (JSCEarlyWarningSystemTest.test_partially_effective_patch): Patch fixes some failures but adds no new failures.
        (JSCEarlyWarningSystemTest.test_different_test_failures): Patch that has some failures not present in tree.
        * Scripts/webkitpy/common/net/jsctestresults_unittest.py: Added.
        (JSCTestResultsTest): Class to test JSCTestResults.
        (JSCTestResultsTest.test_results_from_string): Tests parsing.
        * Scripts/webkitpy/tool/bot/jsctestresultsreader.py: Added.
        (JSCTestResultsReader): Reads results file.
        (JSCTestResultsReader.__init__): Sets attributes.
        (JSCTestResultsReader._read_file_contents): Reads file.
        (JSCTestResultsReader._create_jsc_test_results): Creates a JSCTestResults object.
        (JSCTestResultsReader.results): Invokes _create_jsc_test_results to return a JSCTestResults object.
        * Scripts/webkitpy/tool/bot/patchanalysistask.py:
        (PatchAnalysisTask._build): Check for JSC.
        (PatchAnalysisTask._build_without_patch): Check for JSC.
        (PatchAnalysisTask._test): Check for JSC.
        (PatchAnalysisTask._build_and_test_without_patch): Check for JSC.
        (PatchAnalysisTask._test_patch): Add retry logic for JSC.
        * Scripts/webkitpy/tool/commands/earlywarningsystem.py:
        (AbstractEarlyWarningSystem.begin_work_queue): Use JSCTestResultsReader instead of LayoutTestResultsReader in JSC EWS.
        (AbstractEarlyWarningSystem.test_results): _layout_test_results_reader -> _test_results_reader.
        (AbstractEarlyWarningSystem.archive_last_test_results): _layout_test_results_reader -> _test_results_reader.
        (AbstractEarlyWarningSystem.group): This attribute determines the type of EWS (eg. JSC).
        (AbstractEarlyWarningSystem.load_ews_classes): Add _group.
        * Scripts/webkitpy/tool/commands/earlywarningsystem_unittest.py:
        (EarlyWarningSystemTest._default_expected_logs): Add _group.
        * Scripts/webkitpy/tool/steps/build.py:
        (Build.options): Check for JSC.
        (Build.build): Check for JSC.
        (Build.run): Check for JSC.
        * Scripts/webkitpy/tool/steps/options.py:
        (Options): Add --group command line option.
        * Scripts/webkitpy/tool/steps/runtests.py:
        (RunTests.options): Add group.
        (RunTests.run): Check for JSC.
        (RunTests._run_webkit_tests): Check for JSC.
        (RunTests): Add _group attribute.
        (RunTests._run_javascriptcore_tests): New.
        * Scripts/webkitpy/tool/steps/steps_unittest.py: Change expected command for running jsc tests.

2016-09-25  Gyuyoung Kim  <gyuyoung.kim@webkit.org>

        [EFL] Adjust EFL indentation to EFL MiniBrowser

Lines 50-54 a/Tools/Scripts/webkitpy/common/config/ews.json_sec1
- a/Tools/Scripts/webkitpy/common/config/ews.json +6 lines
50	"port": "mac",	50	"port": "mac",
51	"name": "mac-32bit-ews",	51	"name": "mac-32bit-ews",
52	"architecture": "i386"	52	"architecture": "i386"
		53	},
		54	"JSC EWS": {
		55	"port": "mac",
		56	"name": "jsc-ews",
		57	"group": "jsc",
		58	"runTests": true
53	}	59	}
54	}	60	}



# Copyright (C) 2009, Google Inc. All rights reserved.
# Copyright (C) 2013 Nokia Corporation and/or its subsidiary(-ies).
# Copyright (C) 2015-2016 Apple Inc. All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are

            command.append("--release")
        return command

    def build_jsc_command(self, build_style=None):
        command = self.script_shell_command("build-jsc")
        if build_style == "debug":
            command.append("--debug")
        return command

    def run_javascriptcore_tests_command(self, build_style=None):
        command = self.script_shell_command("run-javascriptcore-tests")
        command.append("--no-fail-fast")
        command.append("--json-output=jsc_test_results.json")
        if (build_style == "debug"):
            command.append("--debug")
        return command

    def run_webkit_tests_command(self, build_style=None):
        command = self.script_shell_command("run-webkit-tests")

class MacPort(DeprecatedPort):
    port_flag_name = "mac"

    def run_webkit_tests_command(self, build_style=None, test_suite=None):
        if (test_suite == "jsc"):
            return super(MacPort, self).run_javascriptcore_tests_command(build_style)

        command = super(MacPort, self).run_webkit_tests_command(build_style)
        command.append("--dump-render-tree")
        return command



# Copyright (C) 2016 Apple Inc. All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions
# are met:
# 1.  Redistributions of source code must retain the above copyright
#     notice, this list of conditions and the following disclaimer.
# 2.  Redistributions in binary form must reproduce the above copyright
#     notice, this list of conditions and the following disclaimer in the
#     documentation and/or other materials provided with the distribution.
#
# THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS'' AND
# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
# DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS BE LIABLE FOR
# ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
# SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

import json
import logging

_log = logging.getLogger(__name__)


class JSCTestResults(object):
    @classmethod
    def results_from_string(cls, string):
        if not string:
            return None
        parsed_results = json.loads(string)

        return cls(parsed_results['allApiTestsPassed'], parsed_results['stressTestFailures'])

    def __init__(self, all_api_tests_passed, stress_test_failures):
        self._all_api_tests_passed = all_api_tests_passed
        self._stress_test_failures = stress_test_failures

        self._test_results = stress_test_failures
        if not self._all_api_tests_passed:
            self._test_results.append("apiTests")

    def __eq__(self, other):
        return self._all_api_tests_passed == other._all_api_tests_passed and set(self._stress_test_failures) == set(other._stress_test_failures)

    def __le__(self, other):
        return set(self.test_results()) <= set(other.test_results())

    def stress_test_failures(self):
        return self._stress_test_failures

    def all_api_tests_passed(self):
        return self._all_api_tests_passed

    def all_passed(self):
        return self._all_api_tests_passed and not self._stress_test_failures

    def failing_test_results(self):
        return self.stress_test_failures()

    def test_results(self):
        return self._test_results

    def failing_tests(self):
        return self._test_results

    # No defined failure limit for JSC.
    def did_exceed_test_failure_limit(self):
        return False



# Copyright (C) 2016 Apple Inc. All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions
# are met:
# 1.  Redistributions of source code must retain the above copyright
#     notice, this list of conditions and the following disclaimer.
# 2.  Redistributions in binary form must reproduce the above copyright
#     notice, this list of conditions and the following disclaimer in the
#     documentation and/or other materials provided with the distribution.
#
# THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS'' AND
# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
# DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS BE LIABLE FOR
# ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
# SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

import unittest

from webkitpy.common.net.jsctestresults import JSCTestResults


class JSCTestResultsTest(unittest.TestCase):
    def test_results_from_string(self):
        no_failures_string = '{"allApiTestsPassed": true, "stressTestFailures":[]}'
        no_failures_results = JSCTestResults(True, [])
        self.assertEqual(no_failures_results, JSCTestResults.results_from_string(no_failures_string))

        api_test_failures_string = '{"allApiTestsPassed": false, "stressTestFailures":[]}'
        api_test_failures_results = JSCTestResults(False, [])
        self.assertEqual(api_test_failures_results, JSCTestResults.results_from_string(api_test_failures_string))

        many_failures_string = '{"allApiTestsPassed": false, "stressTestFailures":["es6.yaml/es6/typed_arrays_Int16Array.js.default", "es6.yaml/es6/typed_arrays_Int8Array.js.default"]}'
        many_failures_results = JSCTestResults(False, ["es6.yaml/es6/typed_arrays_Int16Array.js.default", "es6.yaml/es6/typed_arrays_Int8Array.js.default"])
        self.assertEqual(many_failures_results, JSCTestResults.results_from_string(many_failures_string))



# Copyright (C) 2017 Apple Inc. All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions
# are met:
# 1.  Redistributions of source code must retain the above copyright
#     notice, this list of conditions and the following disclaimer.
# 2.  Redistributions in binary form must reproduce the above copyright
#     notice, this list of conditions and the following disclaimer in the
#     documentation and/or other materials provided with the distribution.
#
# THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS'' AND
# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
# DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS BE LIABLE FOR
# ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
# SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

import logging
import unittest

from webkitpy.common.net.jsctestresults import JSCTestResults
from webkitpy.common.system.executive import ScriptError
from webkitpy.tool.bot.patchanalysistask import *
from webkitpy.tool.commands.earlywarningsystem import AbstractEarlyWarningSystem
from webkitpy.tool.mocktool import MockTool

_log = logging.getLogger(__name__)


class MockPatchAnalysisTask(PatchAnalysisTask):
    def __init__(self, delegate, patch, first_test_all_passed):
        self._delegate = delegate
        self._patch = patch
        self._script_error = None
        self._results_archive_from_patch_test_run = None
        self._results_from_patch_test_run = None
        self.error = None
        self._first_test_all_passed = first_test_all_passed
        self._test_run_count = 0

    def _test(self):
        self._test_run_count += 1

        if self._first_test_all_passed == True:
            return True
        else:
            self._script_error = ScriptError("Expected failure in regression test")
            return False

    def _build_and_test_without_patch(self):
        self._test_run_count += 1
        return True

    def validate(self):
        return True

    def test_run_count(self):
        return self._test_run_count


# This is the delegate to be used with MockPatchAnalysisTask, above.
class MockJSCEarlyWarningSystem(AbstractEarlyWarningSystem):
    def __init__(self, first_test_results, clean_test_results):
        AbstractEarlyWarningSystem.__init__(self)
        self._group = "jsc"
        self._results_in_order = [clean_test_results, first_test_results]

    def test_results(self):
        return self._results_in_order.pop()


class JSCEarlyWarningSystemTest(unittest.TestCase):
    def _create_task(self, first_test_results, clean_test_results):
        queue = MockJSCEarlyWarningSystem(first_test_results, clean_test_results)
        tool = MockTool(log_executive=True)
        patch = tool.bugs.fetch_attachment(10000)
        return MockPatchAnalysisTask(queue, patch, first_test_results.all_passed())

    def test_success_case(self):
        first_test_results = JSCTestResults(True, [])
        clean_test_results = JSCTestResults(True, [])
        task = self._create_task(first_test_results, clean_test_results)

        return_value = task._test_patch()
        self.assertEqual(task.test_run_count(), 1)
        self.assertEqual(return_value, True)

    def test_test_failure(self):
        first_test_results = JSCTestResults(True, ['Fail.js'])
        clean_test_results = JSCTestResults(True, [])
        task = self._create_task(first_test_results, clean_test_results)

        with self.assertRaises(ScriptError):
            return_value = task._test_patch()
        self.assertEqual(task.test_run_count(), 2)

    def test_fix(self):
        first_test_results = JSCTestResults(True, [])
        clean_test_results = JSCTestResults(True, ['Fail.js'])
        task = self._create_task(first_test_results, clean_test_results)

        return_value = task._test_patch()
        self.assertEqual(task.test_run_count(), 1)
        self.assertEqual(return_value, True)

    def test_ineffective_patch(self):
        first_test_results = JSCTestResults(False, ['failure1.js', 'failure2.js'])
        clean_test_results = JSCTestResults(False, ['failure1.js', 'failure2.js'])
        task = self._create_task(first_test_results, clean_test_results)

        return_value = task._test_patch()
        self.assertEqual(task.test_run_count(), 2)
        self.assertEqual(return_value, True)

    def test_partially_effective_patch(self):
        first_test_results = JSCTestResults(True, ['failure2.js'])
        clean_test_results = JSCTestResults(False, ['failure1.js', 'failure2.js'])
        task = self._create_task(first_test_results, clean_test_results)

        return_value = task._test_patch()
        self.assertEqual(task.test_run_count(), 2)
        self.assertEqual(return_value, True)

    def test_different_test_failures(self):
        first_test_results = JSCTestResults(False, [])
        clean_test_results = JSCTestResults(True, ['failure1.js', 'failure2.js'])
        task = self._create_task(first_test_results, clean_test_results)

        with self.assertRaises(ScriptError):
            return_value = task._test_patch()
        self.assertEqual(task.test_run_count(), 2)



# Copyright (C) 2016 Apple Inc. All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions
# are met:
# 1.  Redistributions of source code must retain the above copyright
#     notice, this list of conditions and the following disclaimer.
# 2.  Redistributions in binary form must reproduce the above copyright
#     notice, this list of conditions and the following disclaimer in the
#     documentation and/or other materials provided with the distribution.
#
# THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS'' AND
# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
# DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS BE LIABLE FOR
# ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
# SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

import logging

from webkitpy.common.net.jsctestresults import JSCTestResults

_log = logging.getLogger(__name__)


class JSCTestResultsReader(object):
    def __init__(self, host, results_directory):
        self._host = host
        self._results_directory = results_directory

    def _read_file_contents(self, path):
        try:
            return self._host.filesystem.read_text_file(path)
        except IOError, e:  # File does not exist or can't be read.
            return None

    def _create_jsc_test_results(self):
        results_path = self._host.filesystem.join(self._results_directory, "jsc_test_results.json")
        contents = self._read_file_contents(results_path)
        return JSCTestResults.results_from_string(contents)

    def results(self):
        return self._create_jsc_test_results()



# Copyright (c) 2010 Google Inc. All rights reserved.
# Copyright (c) 2016 Apple Inc. All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are

        "Patch does not apply")

    def _build(self):
        args = [
            "build",
            "--no-clean",
            "--no-update",
            "--build-style=%s" % self._delegate.build_style(),
        ]

        if hasattr(self._delegate, 'group'):
            args.append("--group=%s" % self._delegate.group())

        return self._run_command(
            args,
            "Built patch",
            "Patch does not build")

    def _build_without_patch(self):
        args = [
            "build",
            "--force-clean",
            "--no-update",
            "--build-style=%s" % self._delegate.build_style(),
        ]

        if hasattr(self._delegate, 'group'):
            args.append("--group=%s" % self._delegate.group())

        return self._run_command(
            args,
            "Able to build without patch",
            "Unable to build without patch")

    def _test(self):
        args = [
            "build-and-test",
            "--no-clean",
            "--no-update",

            "--test",
            "--non-interactive",
            "--build-style=%s" % self._delegate.build_style(),
        ]

        if hasattr(self._delegate, 'group'):
            args.append("--group=%s" % self._delegate.group())

        return self._run_command(
            args,
            "Passed tests",
            "Patch does not pass tests")

    def _build_and_test_without_patch(self):
        args = [
            "build-and-test",
            "--force-clean",
            "--no-update",

            "--test",
            "--non-interactive",
            "--build-style=%s" % self._delegate.build_style(),
        ]

        if hasattr(self._delegate, 'group'):
            args.append("--group=%s" % self._delegate.group())

        return self._run_command(
            args,
            "Able to pass tests without patch",
            "Unable to pass tests without patch (tree is red?)")

    def _land(self):
        # Unclear if this should pass --quiet or not.  If --parent-command always does the reporting, then it should.

        if self._test():
            return True

        # FIXME: Abstract out the retry logic instead of casing on the test suite here.
        if hasattr(self._delegate, 'group') and self._delegate.group() == "jsc":
            first_results = self._delegate.test_results()
            first_script_error = self._script_error
            if first_results.all_passed():
                return True

        if self._test() and not first_results.did_exceed_test_failure_limit():
            # Only report flaky tests if we were successful at parsing results.json and archiving results.
            if first_results and first_results_archive:
                self._report_flaky_tests(first_results.failing_test_results(), first_results_archive)
            return True

        second_results = self._delegate.test_results()
        second_results_archive = self._delegate.archive_last_test_results(self._patch)
        second_script_error = self._script_error
        second_failure_status_id = self.failure_status_id

        if second_results.did_exceed_test_failure_limit() and first_results.did_exceed_test_failure_limit():
            self._build_and_test_without_patch()
            clean_tree_results = self._delegate.test_results()

            # If the patch's failures are a subset of the tree's failures, the patch should pass EWS.
            if first_results <= clean_tree_results:
                return True

            return self.report_failure(None, first_results, first_script_error)

        else:
            # Note: archive_last_test_results deletes the results directory, making these calls order-sensitve.
            # We could remove this dependency by building the test_results from the archive.
            first_results = self._delegate.test_results()
            first_results_archive = self._delegate.archive_last_test_results(self._patch)
            first_script_error = self._script_error
            first_failure_status_id = self.failure_status_id

            if self._test() and not first_results.did_exceed_test_failure_limit():
                # Only report flaky tests if we were successful at parsing results.json and archiving results.
                if first_results and first_results_archive:
                    self._report_flaky_tests(first_results.failing_test_results(), first_results_archive)
                return True

            second_results = self._delegate.test_results()
            second_results_archive = self._delegate.archive_last_test_results(self._patch)
            second_script_error = self._script_error
            second_failure_status_id = self.failure_status_id

            if second_results.did_exceed_test_failure_limit() and first_results.did_exceed_test_failure_limit():
                self._build_and_test_without_patch()
                clean_tree_results = self._delegate.test_results()

                if (len(first_results.failing_tests()) - len(clean_tree_results.failing_tests())) <= 5:
                    return False

                self.failure_status_id = first_failure_status_id
            self._report_flaky_tests(tests_that_only_failed_second, second_results_archive)

                return self.report_failure(first_results_archive, first_results, first_script_error)
            if tests_that_consistently_failed:
                if self._should_defer_patch_or_throw(tests_that_consistently_failed, first_results_archive, first_script_error, first_failure_status_id):
                    return False  # Defer patch

            if second_results.did_exceed_test_failure_limit():
                self._should_defer_patch_or_throw(first_results.failing_test_results(), first_results_archive, first_script_error, first_failure_status_id)
                return False

            if first_results.did_exceed_test_failure_limit():
                self._should_defer_patch_or_throw(second_results.failing_test_results(), second_results_archive, second_script_error, second_failure_status_id)
                return False

            if self._results_failed_different_tests(first_results, second_results):
                first_failing_results_set = frozenset(first_results.failing_test_results())
                second_failing_results_set = frozenset(second_results.failing_test_results())

                tests_that_only_failed_first = first_failing_results_set.difference(second_failing_results_set)
                self._report_flaky_tests(tests_that_only_failed_first, first_results_archive)

                tests_that_only_failed_second = second_failing_results_set.difference(first_failing_results_set)
                self._report_flaky_tests(tests_that_only_failed_second, second_results_archive)

                tests_that_consistently_failed = first_failing_results_set.intersection(second_failing_results_set)
                if tests_that_consistently_failed:
                    if self._should_defer_patch_or_throw(tests_that_consistently_failed, first_results_archive, first_script_error, first_failure_status_id):
                        return False  # Defer patch

                # At this point we know that at least one test flaked, but no consistent failures
                # were introduced. This is a bit of a grey-zone.
                return False  # Defer patch

            if self._should_defer_patch_or_throw(first_results.failing_test_results(), first_results_archive, first_script_error, first_failure_status_id):
                return False  # Defer patch

            # At this point, we know that the first and second runs had the exact same failures,
            # and that those failures are all present on the clean tree, so we can say with certainty
            # that the patch is good.
            return True

    def results_archive_from_patch_test_run(self, patch):
        assert(self._patch.id() == patch.id())  # PatchAnalysisTask is not currently re-useable.

Lines 1-4 a/Tools/Scripts/webkitpy/tool/commands/earlywarningsystem.py_sec1
- a/Tools/Scripts/webkitpy/tool/commands/earlywarningsystem.py -3 / +15 lines
1	# Copyright (c) 2009 Google Inc. All rights reserved.	1	# Copyright (c) 2009 Google Inc. All rights reserved.
		2	# Copyright (c) 2016 Apple Inc. All rights reserved.
2	#	3	#
3	# Redistribution and use in source and binary forms, with or without	4	# Redistribution and use in source and binary forms, with or without
4	# modification, are permitted provided that the following conditions are	5	# modification, are permitted provided that the following conditions are
Lines 28-33 a/Tools/Scripts/webkitpy/tool/commands/earlywarningsystem.py_sec2
28		29
29	import json	30	import json
30	import logging	31	import logging
		32	import os
31	from optparse import make_option	33	from optparse import make_option
32		34
33	from webkitpy.common.config.committers import CommitterList	35	from webkitpy.common.config.committers import CommitterList
Lines 36-41 from webkitpy.common.system.filesystem import FileSystem a/Tools/Scripts/webkitpy/tool/commands/earlywarningsystem.py_sec3
36	from webkitpy.common.system.executive import ScriptError	38	from webkitpy.common.system.executive import ScriptError
37	from webkitpy.tool.bot.earlywarningsystemtask import EarlyWarningSystemTask, EarlyWarningSystemTaskDelegate	39	from webkitpy.tool.bot.earlywarningsystemtask import EarlyWarningSystemTask, EarlyWarningSystemTaskDelegate
38	from webkitpy.tool.bot.layouttestresultsreader import LayoutTestResultsReader	40	from webkitpy.tool.bot.layouttestresultsreader import LayoutTestResultsReader
		41	from webkitpy.tool.bot.jsctestresultsreader import JSCTestResultsReader
39	from webkitpy.tool.bot.patchanalysistask import UnableToApplyPatch, PatchIsNotValid	42	from webkitpy.tool.bot.patchanalysistask import UnableToApplyPatch, PatchIsNotValid
40	from webkitpy.tool.bot.queueengine import QueueEngine	43	from webkitpy.tool.bot.queueengine import QueueEngine
41	from webkitpy.tool.commands.queues import AbstractReviewQueue	44	from webkitpy.tool.commands.queues import AbstractReviewQueue
Lines 53-59 class AbstractEarlyWarningSystem(AbstractReviewQueue, EarlyWarningSystemTaskDele a/Tools/Scripts/webkitpy/tool/commands/earlywarningsystem.py_sec4
53		56
54	def begin_work_queue(self):	57	def begin_work_queue(self):
55	AbstractReviewQueue.begin_work_queue(self)	58	AbstractReviewQueue.begin_work_queue(self)
56	self._layout_test_results_reader = LayoutTestResultsReader(self._tool, self._port.results_directory(), self._log_directory())	59
		60	if self.group() == "jsc":
		61	dir = os.path.dirname(os.path.dirname(os.path.dirname(self._port.results_directory())))
		62	self._test_results_reader = JSCTestResultsReader(self._tool, dir)
		63	else:
		64	self._test_results_reader = LayoutTestResultsReader(self._tool, self._port.results_directory(), self._log_directory())
57		65
58	def _failing_tests_message(self, task, patch):	66	def _failing_tests_message(self, task, patch):
59	results = task.results_from_patch_test_run(patch)	67	results = task.results_from_patch_test_run(patch)
Lines 117-130 class AbstractEarlyWarningSystem(AbstractReviewQueue, EarlyWarningSystemTaskDele a/Tools/Scripts/webkitpy/tool/commands/earlywarningsystem.py_sec5
117	return self._update_status(message, patch=patch, results_file=failure_log)	125	return self._update_status(message, patch=patch, results_file=failure_log)
118		126
119	def test_results(self):	127	def test_results(self):
120	return self._layout_test_results_reader.results()	128	return self._test_results_reader.results()
121		129
122	def archive_last_test_results(self, patch):	130	def archive_last_test_results(self, patch):
123	return self._layout_test_results_reader.archive(patch)	131	return self._test_results_reader.archive(patch)
124		132
125	def build_style(self):	133	def build_style(self):
126	return self._build_style	134	return self._build_style
127		135
		136	def group(self):
		137	return self._group
		138
128	def refetch_patch(self, patch):	139	def refetch_patch(self, patch):
129	return self._tool.bugs.fetch_attachment(patch.id())	140	return self._tool.bugs.fetch_attachment(patch.id())
130		141
Lines 156-160 class AbstractEarlyWarningSystem(AbstractReviewQueue, EarlyWarningSystemTaskDele a/Tools/Scripts/webkitpy/tool/commands/earlywarningsystem.py_sec6
156	'_build_style': config.get('style', "release"),	167	'_build_style': config.get('style', "release"),
157	'watchers': config.get('watchers', []),	168	'watchers': config.get('watchers', []),
158	'run_tests': config.get('runTests', cls.run_tests),	169	'run_tests': config.get('runTests', cls.run_tests),
		170	'_group': config.get('group', None),
159	}))	171	}))
160	return classes	172	return classes



# Copyright (C) 2009 Google Inc. All rights reserved.
# Copyright (C) 2016 Apple Inc. All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are

        class TestEWS(AbstractEarlyWarningSystem):
            port_name = "win"  # Needs to be a port which port/factory understands.
            _build_style = None
            _group = None

        ews = TestEWS()
        ews.bind_to_tool(MockTool())

            "port": ews.port_name,
            "architecture": " --architecture=%s" % ews.architecture if ews.architecture else "",
            "build_style": ews.build_style(),
            "group": ews.group(),
        }
        if ews.run_tests:
            run_tests_line = "Running: webkit-patch --status-host=example.com build-and-test --no-clean --no-update --test --non-interactive --build-style=%(build_style)s --group=%(group)s --port=%(port)s%(architecture)s\n" % string_replacements
        else:
            run_tests_line = ""
        string_replacements['run_tests_line'] = run_tests_line

Running: webkit-patch --status-host=example.com clean --port=%(port)s%(architecture)s
Running: webkit-patch --status-host=example.com update --port=%(port)s%(architecture)s
Running: webkit-patch --status-host=example.com apply-attachment --no-update --non-interactive 10000 --port=%(port)s%(architecture)s
Running: webkit-patch --status-host=example.com build --no-clean --no-update --build-style=%(build_style)s --group=%(group)s --port=%(port)s%(architecture)s
%(run_tests_line)sMOCK: update_status: %(name)s Pass
MOCK: release_work_item: %(name)s 10000
""" % string_replacements,



# Copyright (C) 2010 Google Inc. All rights reserved.
# Copyright (C) 2016 Apple Inc. All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are

            Options.build,
            Options.quiet,
            Options.build_style,
            Options.group,
        ]

    def build(self, build_style, group):
        environment = self._tool.copy_current_environment()
        environment.disable_gcc_smartquotes()
        environment.disable_jhbuild_VT100_output()
        env = environment.to_dictionary()

        if group == "jsc":
            build_command = self._tool.deprecated_port().build_jsc_command(build_style=build_style)
        else:
            build_command = self._tool.deprecated_port().build_webkit_command(build_style=build_style)

        if self._options.architecture:
            build_command += ['ARCHS=%s' % self._options.architecture]

        self._tool.executive.run_and_throw_if_fail(build_command, self._options.quiet,
            cwd=self._tool.scm().checkout_root, env=env)

    def run(self, state):
        if not self._options.build:
            return
        _log.info("Building WebKit")

        group = None
        if hasattr(self._options, 'group'):
            group = self._options.group

        if self._options.build_style == "both":
            self.build("debug", group)
            self.build("release", group)
        else:
            self.build(self._options.build_style, group)



# Copyright (C) 2010 Google Inc. All rights reserved.
# Copyright (C) 2016 Apple Inc. All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are

    email = make_option("--email", action="store", type="string", dest="email", help="Email address to use in ChangeLogs.")
    force_clean = make_option("--force-clean", action="store_true", dest="force_clean", default=False, help="Clean working directory before applying patches (removes local changes and commits)")
    git_commit = make_option("-g", "--git-commit", action="store", dest="git_commit", help="Operate on a local commit. If a range, the commits are squashed into one. <ref>.... includes the working copy changes. UPSTREAM can be used for the upstream/tracking branch.")
    group = make_option("--group", action="store", dest="group", default=None, help="")
    local_commit = make_option("--local-commit", action="store_true", dest="local_commit", default=False, help="Make a local commit for each applied patch")
    non_interactive = make_option("--non-interactive", action="store_true", dest="non_interactive", default=False, help="Never prompt the user, fail as fast as possible.")
    obsolete_patches = make_option("--no-obsolete", action="store_false", dest="obsolete_patches", default=True, help="Do not obsolete old patches before posting this one.")

Lines 1-4 a/Tools/Scripts/webkitpy/tool/steps/runtests.py_sec1
- a/Tools/Scripts/webkitpy/tool/steps/runtests.py +10 lines
1	# Copyright (C) 2010 Google Inc. All rights reserved.	1	# Copyright (C) 2010 Google Inc. All rights reserved.
		2	# Copyright (C) 2016 Apple Inc. All rights reserved.
2	#	3	#
3	# Redistribution and use in source and binary forms, with or without	4	# Redistribution and use in source and binary forms, with or without
4	# modification, are permitted provided that the following conditions are	5	# modification, are permitted provided that the following conditions are
Lines 53-58 class RunTests(AbstractStep): a/Tools/Scripts/webkitpy/tool/steps/runtests.py_sec2
53	Options.iterate_on_new_tests,	54	Options.iterate_on_new_tests,
54	Options.non_interactive,	55	Options.non_interactive,
55	Options.quiet,	56	Options.quiet,
		57	Options.group,
56	]	58	]
57		59
58	def run(self, state):	60	def run(self, state):
Lines 63-68 class RunTests(AbstractStep): a/Tools/Scripts/webkitpy/tool/steps/runtests.py_sec3
63	if not self._options.test:	65	if not self._options.test:
64	return	66	return
65		67
		68	if hasattr(self._options, "group") and self._options.group == "jsc":
		69	self._run_javascriptcore_tests()
		70	return
		71
66	python_unittests_command = self._tool.deprecated_port().run_python_unittests_command()	72	python_unittests_command = self._tool.deprecated_port().run_python_unittests_command()
67	if python_unittests_command:	73	if python_unittests_command:
68	_log.info("Running Python unit tests")	74	_log.info("Running Python unit tests")
Lines 143-145 class RunTests(AbstractStep): a/Tools/Scripts/webkitpy/tool/steps/runtests.py_sec4
143	args.append("--iterations=%d" % iterate_on_new_tests)	149	args.append("--iterations=%d" % iterate_on_new_tests)
144		150
145	self._tool.executive.run_and_throw_if_fail(args, cwd=self._tool.scm().checkout_root)	151	self._tool.executive.run_and_throw_if_fail(args, cwd=self._tool.scm().checkout_root)
		152
		153	def _run_javascriptcore_tests(self):
		154	args = self._tool.deprecated_port().run_javascriptcore_tests_command(self._options.build_style)
		155	self._tool.executive.run_and_throw_if_fail(args, cwd=self._tool.scm().checkout_root)



# Copyright (C) 2010 Google Inc. All rights reserved.
# Copyright (C) 2016 Apple Inc. All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are

        self._assert_step_output_with_bug(steps.EnsureBugIsOpenAndAssigned, 50002, expected_logs)

    def test_runtests_args(self):
        self.maxDiff = None
        mock_options = self._step_options()
        mock_options.non_interactive = False
        mock_options.build_style = "release"

Running Perl unit tests
MOCK run_and_throw_if_fail: ['Tools/Scripts/test-webkitperl'], cwd=/mock-checkout
Running JavaScriptCore tests
MOCK run_and_throw_if_fail: ['Tools/Scripts/run-javascriptcore-tests', '--no-fail-fast', '--json-output=jsc_test_results.json'], cwd=/mock-checkout
Running bindings generation tests
MOCK run_and_throw_if_fail: ['Tools/Scripts/run-bindings-tests'], cwd=/mock-checkout
Running run-webkit-tests

        OutputCapture().assert_outputs(self, step.run, [{}], expected_logs=expected_logs)

    def test_runtests_debug_args(self):
        self.maxDiff = None
        mock_options = self._step_options()
        mock_options.non_interactive = False
        mock_options.build_style = "debug"

Running Perl unit tests
MOCK run_and_throw_if_fail: ['Tools/Scripts/test-webkitperl'], cwd=/mock-checkout
Running JavaScriptCore tests
MOCK run_and_throw_if_fail: ['Tools/Scripts/run-javascriptcore-tests', '--no-fail-fast', '--json-output=jsc_test_results.json'], cwd=/mock-checkout
Running bindings generation tests
MOCK run_and_throw_if_fail: ['Tools/Scripts/run-bindings-tests'], cwd=/mock-checkout
Running run-webkit-tests

Return to Bug 162458

Lines 1-3 a/Tools/ChangeLog_sec1
- a/Tools/ChangeLog +79 lines
		1	2017-01-12 Srinivasan Vijayaraghavan <svijayaraghavan@apple.com>
		2
		3	EWS should run JavaScriptCore tests
		4	https://bugs.webkit.org/show_bug.cgi?id=162458
		5
		6	Reviewed by NOBODY (OOPS!).
		7
		8	* Scripts/webkitpy/common/config/ews.json: Add config info for JSC EWS.
		9	* Scripts/webkitpy/common/config/ports.py: Add support for JSC EWS in Mac Port.
		10	(DeprecatedPort.build_jsc_command): Added - command to build JSC only (quicker than building all of WebKit).
		11	(DeprecatedPort.run_javascriptcore_tests_command): Allow JSC EWS to only run JSC tests.
		12	(MacPort.run_webkit_tests_command): Check for JSC.
		13	* Scripts/webkitpy/common/net/jsctestresults.py: Added.
		14	(JSCTestResults):
		15	(JSCTestResults.results_from_string): Use json library to parse results.
		16	(JSCTestResults.__init__): Initialize members.
		17	(JSCTestResults.__eq__): This enables unit testing.
		18	(JSCTestResults.stress_test_failures): Getter.
		19	(JSCTestResults.all_api_tests_passed): Getter.
		20	(JSCTestResults.all_passed): Getter.
		21	(JSCTestResults.failing_test_results): Getter. Some of these exist for interopability reasons.
		22	(JSCTestResults.test_results): Getter.
		23	(JSCTestResults.failing_tests): Getter.
		24	(JSCTestResults.did_exceed_test_failure_limit): Getter.
		25	* Scripts/webkitpy/tool/bot/jscews_unittest.py: Added.
		26	(MockPatchAnalysisTask): Mocked-out version of PatchAnalysisTask that doesn't run commands.
		27	(MockPatchAnalysisTask.__init__): Sets attributes.
		28	(MockPatchAnalysisTask._test): Override to not run command.
		29	(MockPatchAnalysisTask._build_and_test_without_patch): Override to not run command.
		30	(MockPatchAnalysisTask.validate): Assume mocked patch is valid for purposes of testing retry logic.
		31	(MockPatchAnalysisTask.test_run_count): Specific to the mocked version, to test retry logic.
		32	(MockJSCEarlyWarningSystem): Mocked-out version of AbstarctEarlyWarningSystem so we can provide test results.
		33	(MockJSCEarlyWarningSystem.__init__): Sets attributes, also sets group to jsc.
		34	(MockJSCEarlyWarningSystem.test_results): Returns test results provided by us, instead of using a JSON reader.
		35	(JSCEarlyWarningSystemTest): Class to test retry logic in below situations.
		36	(JSCEarlyWarningSystemTest._create_task): Helper function to abstract out common code.
		37	(JSCEarlyWarningSystemTest.test_success_case): Clean patch on clean tree.
		38	(JSCEarlyWarningSystemTest.test_test_failure): Failed patch on clean tree.
		39	(JSCEarlyWarningSystemTest.test_fix): Patch that fixes all tree redness.
		40	(JSCEarlyWarningSystemTest.test_ineffective_patch): Patch that has same failures as tree.
		41	(JSCEarlyWarningSystemTest.test_partially_effective_patch): Patch fixes some failures but adds no new failures.
		42	(JSCEarlyWarningSystemTest.test_different_test_failures): Patch that has some failures not present in tree.
		43	* Scripts/webkitpy/common/net/jsctestresults_unittest.py: Added.
		44	(JSCTestResultsTest): Class to test JSCTestResults.
		45	(JSCTestResultsTest.test_results_from_string): Tests parsing.
		46	* Scripts/webkitpy/tool/bot/jsctestresultsreader.py: Added.
		47	(JSCTestResultsReader): Reads results file.
		48	(JSCTestResultsReader.__init__): Sets attributes.
		49	(JSCTestResultsReader._read_file_contents): Reads file.
		50	(JSCTestResultsReader._create_jsc_test_results): Creates a JSCTestResults object.
		51	(JSCTestResultsReader.results): Invokes _create_jsc_test_results to return a JSCTestResults object.
		52	* Scripts/webkitpy/tool/bot/patchanalysistask.py:
		53	(PatchAnalysisTask._build): Check for JSC.
		54	(PatchAnalysisTask._build_without_patch): Check for JSC.
		55	(PatchAnalysisTask._test): Check for JSC.
		56	(PatchAnalysisTask._build_and_test_without_patch): Check for JSC.
		57	(PatchAnalysisTask._test_patch): Add retry logic for JSC.
		58	* Scripts/webkitpy/tool/commands/earlywarningsystem.py:
		59	(AbstractEarlyWarningSystem.begin_work_queue): Use JSCTestResultsReader instead of LayoutTestResultsReader in JSC EWS.
		60	(AbstractEarlyWarningSystem.test_results): _layout_test_results_reader -> _test_results_reader.
		61	(AbstractEarlyWarningSystem.archive_last_test_results): _layout_test_results_reader -> _test_results_reader.
		62	(AbstractEarlyWarningSystem.group): This attribute determines the type of EWS (eg. JSC).
		63	(AbstractEarlyWarningSystem.load_ews_classes): Add _group.
		64	* Scripts/webkitpy/tool/commands/earlywarningsystem_unittest.py:
65	(EarlyWarningSystemTest._default_expected_logs): Add _group.
66	* Scripts/webkitpy/tool/steps/build.py:
67	(Build.options): Check for JSC.
68	(Build.build): Check for JSC.
69	(Build.run): Check for JSC.
70	* Scripts/webkitpy/tool/steps/options.py:
71	(Options): Add --group command line option.
72	* Scripts/webkitpy/tool/steps/runtests.py:
73	(RunTests.options): Add group.
74	(RunTests.run): Check for JSC.
75	(RunTests._run_webkit_tests): Check for JSC.
76	(RunTests): Add _group attribute.
77	(RunTests._run_javascriptcore_tests): New.
78	* Scripts/webkitpy/tool/steps/steps_unittest.py: Change expected command for running jsc tests.
79
1	2016-09-25 Gyuyoung Kim <gyuyoung.kim@webkit.org>	80	2016-09-25 Gyuyoung Kim <gyuyoung.kim@webkit.org>
2		81
3	[EFL] Adjust EFL indentation to EFL MiniBrowser	82	[EFL] Adjust EFL indentation to EFL MiniBrowser

Lines 1-6 a/Tools/Scripts/webkitpy/common/config/ports.py_sec1
- a/Tools/Scripts/webkitpy/common/config/ports.py -4 / +18 lines
1	# Copyright (C) 2009, Google Inc. All rights reserved.	1	# Copyright (C) 2009, Google Inc. All rights reserved.
2	# Copyright (C) 2013 Nokia Corporation and/or its subsidiary(-ies).	2	# Copyright (C) 2013 Nokia Corporation and/or its subsidiary(-ies).
3	# Copyright (C) 2015 Apple Inc. All rights reserved.	3	# Copyright (C) 2015-2016 Apple Inc. All rights reserved.
4	#	4	#
5	# Redistribution and use in source and binary forms, with or without	5	# Redistribution and use in source and binary forms, with or without
6	# modification, are permitted provided that the following conditions are	6	# modification, are permitted provided that the following conditions are
Lines 103-110 class DeprecatedPort(object): a/Tools/Scripts/webkitpy/common/config/ports.py_sec2
103	command.append("--release")	103	command.append("--release")
104	return command	104	return command
105		105
106	def run_javascriptcore_tests_command(self):	106	def build_jsc_command(self, build_style=None):
107	return self.script_shell_command("run-javascriptcore-tests")	107	command = self.script_shell_command("build-jsc")
		108	if build_style == "debug":
		109	command.append("--debug")
		110	return command
		111
		112	def run_javascriptcore_tests_command(self, build_style=None):
		113	command = self.script_shell_command("run-javascriptcore-tests")
		114	command.append("--no-fail-fast")
		115	command.append("--json-output=jsc_test_results.json")
		116	if (build_style == "debug"):
		117	command.append("--debug")
		118	return command
108		119
109	def run_webkit_tests_command(self, build_style=None):	120	def run_webkit_tests_command(self, build_style=None):
110	command = self.script_shell_command("run-webkit-tests")	121	command = self.script_shell_command("run-webkit-tests")
Lines 150-156 class IOSSimulatorWK2Port(DeprecatedPort): a/Tools/Scripts/webkitpy/common/config/ports.py_sec3
150	class MacPort(DeprecatedPort):	161	class MacPort(DeprecatedPort):
151	port_flag_name = "mac"	162	port_flag_name = "mac"
152		163
153	def run_webkit_tests_command(self, build_style=None):	164	def run_webkit_tests_command(self, build_style=None, test_suite=None):
		165	if (test_suite == "jsc"):
		166	return super(MacPort, self).run_javascriptcore_tests_command(build_style)
		167
154	command = super(MacPort, self).run_webkit_tests_command(build_style)	168	command = super(MacPort, self).run_webkit_tests_command(build_style)
155	command.append("--dump-render-tree")	169	command.append("--dump-render-tree")
156	return command	170	return command

Line 0 a/Tools/Scripts/webkitpy/common/net/jsctestresults.py_sec1
- a/Tools/Scripts/webkitpy/common/net/jsctestresults.py +72 lines
		1	# Copyright (C) 2016 Apple Inc. All rights reserved.
		2	#
		3	# Redistribution and use in source and binary forms, with or without
		4	# modification, are permitted provided that the following conditions
		5	# are met:
		6	# 1. Redistributions of source code must retain the above copyright
		7	# notice, this list of conditions and the following disclaimer.
		8	# 2. Redistributions in binary form must reproduce the above copyright
		9	# notice, this list of conditions and the following disclaimer in the
		10	# documentation and/or other materials provided with the distribution.
		11	#
		12	# THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS'' AND
		13	# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
		14	# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
		15	# DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS BE LIABLE FOR
		16	# ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
		17	# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
		18	# SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
		19	# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
		20	# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
		21	# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
		22
		23	import json
		24	import logging
		25
		26	_log = logging.getLogger(__name__)
		27
		28
		29	class JSCTestResults(object):
		30	@classmethod
		31	def results_from_string(cls, string):
		32	if not string:
		33	return None
		34	parsed_results = json.loads(string)
		35
		36	return cls(parsed_results['allApiTestsPassed'], parsed_results['stressTestFailures'])
		37
		38	def __init__(self, all_api_tests_passed, stress_test_failures):
		39	self._all_api_tests_passed = all_api_tests_passed
		40	self._stress_test_failures = stress_test_failures
		41
		42	self._test_results = stress_test_failures
		43	if not self._all_api_tests_passed:
		44	self._test_results.append("apiTests")
		45
		46	def __eq__(self, other):
		47	return self._all_api_tests_passed == other._all_api_tests_passed and set(self._stress_test_failures) == set(other._stress_test_failures)
		48
		49	def __le__(self, other):
		50	return set(self.test_results()) <= set(other.test_results())
		51
		52	def stress_test_failures(self):
		53	return self._stress_test_failures
		54
		55	def all_api_tests_passed(self):
		56	return self._all_api_tests_passed
		57
		58	def all_passed(self):
		59	return self._all_api_tests_passed and not self._stress_test_failures
		60
		61	def failing_test_results(self):
		62	return self.stress_test_failures()
		63
		64	def test_results(self):
65	return self._test_results
66
67	def failing_tests(self):
68	return self._test_results
69
70	# No defined failure limit for JSC.
71	def did_exceed_test_failure_limit(self):
72	return False

Line 0 a/Tools/Scripts/webkitpy/tool/bot/jscews_unittest.py_sec1
- a/Tools/Scripts/webkitpy/tool/bot/jscews_unittest.py +136 lines
		1	# Copyright (C) 2017 Apple Inc. All rights reserved.
		2	#
		3	# Redistribution and use in source and binary forms, with or without
		4	# modification, are permitted provided that the following conditions
		5	# are met:
		6	# 1. Redistributions of source code must retain the above copyright
		7	# notice, this list of conditions and the following disclaimer.
		8	# 2. Redistributions in binary form must reproduce the above copyright
		9	# notice, this list of conditions and the following disclaimer in the
		10	# documentation and/or other materials provided with the distribution.
		11	#
		12	# THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS'' AND
		13	# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
		14	# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
		15	# DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS BE LIABLE FOR
		16	# ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
		17	# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
		18	# SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
		19	# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
		20	# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
		21	# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
		22
		23	import logging
		24	import unittest
		25
		26	from webkitpy.common.net.jsctestresults import JSCTestResults
		27	from webkitpy.common.system.executive import ScriptError
		28	from webkitpy.tool.bot.patchanalysistask import *
		29	from webkitpy.tool.commands.earlywarningsystem import AbstractEarlyWarningSystem
		30	from webkitpy.tool.mocktool import MockTool
		31
		32	_log = logging.getLogger(__name__)
		33
		34
		35	class MockPatchAnalysisTask(PatchAnalysisTask):
		36	def __init__(self, delegate, patch, first_test_all_passed):
		37	self._delegate = delegate
		38	self._patch = patch
		39	self._script_error = None
		40	self._results_archive_from_patch_test_run = None
		41	self._results_from_patch_test_run = None
		42	self.error = None
		43	self._first_test_all_passed = first_test_all_passed
		44	self._test_run_count = 0
		45
		46	def _test(self):
		47	self._test_run_count += 1
		48
		49	if self._first_test_all_passed == True:
		50	return True
		51	else:
		52	self._script_error = ScriptError("Expected failure in regression test")
		53	return False
		54
		55	def _build_and_test_without_patch(self):
		56	self._test_run_count += 1
		57	return True
		58
		59	def validate(self):
		60	return True
		61
		62	def test_run_count(self):
		63	return self._test_run_count
		64
65
66	# This is the delegate to be used with MockPatchAnalysisTask, above.
67	class MockJSCEarlyWarningSystem(AbstractEarlyWarningSystem):
68	def __init__(self, first_test_results, clean_test_results):
69	AbstractEarlyWarningSystem.__init__(self)
70	self._group = "jsc"
71	self._results_in_order = [clean_test_results, first_test_results]
72
73	def test_results(self):
74	return self._results_in_order.pop()
75
76
77	class JSCEarlyWarningSystemTest(unittest.TestCase):
78	def _create_task(self, first_test_results, clean_test_results):
79	queue = MockJSCEarlyWarningSystem(first_test_results, clean_test_results)
80	tool = MockTool(log_executive=True)
81	patch = tool.bugs.fetch_attachment(10000)
82	return MockPatchAnalysisTask(queue, patch, first_test_results.all_passed())
83
84	def test_success_case(self):
85	first_test_results = JSCTestResults(True, [])
86	clean_test_results = JSCTestResults(True, [])
87	task = self._create_task(first_test_results, clean_test_results)
88
89	return_value = task._test_patch()
90	self.assertEqual(task.test_run_count(), 1)
91	self.assertEqual(return_value, True)
92
93	def test_test_failure(self):
94	first_test_results = JSCTestResults(True, ['Fail.js'])
95	clean_test_results = JSCTestResults(True, [])
96	task = self._create_task(first_test_results, clean_test_results)
97
98	with self.assertRaises(ScriptError):
99	return_value = task._test_patch()
100	self.assertEqual(task.test_run_count(), 2)
101
102	def test_fix(self):
103	first_test_results = JSCTestResults(True, [])
104	clean_test_results = JSCTestResults(True, ['Fail.js'])
105	task = self._create_task(first_test_results, clean_test_results)
106
107	return_value = task._test_patch()
108	self.assertEqual(task.test_run_count(), 1)
109	self.assertEqual(return_value, True)
110
111	def test_ineffective_patch(self):
112	first_test_results = JSCTestResults(False, ['failure1.js', 'failure2.js'])
113	clean_test_results = JSCTestResults(False, ['failure1.js', 'failure2.js'])
114	task = self._create_task(first_test_results, clean_test_results)
115
116	return_value = task._test_patch()
117	self.assertEqual(task.test_run_count(), 2)
118	self.assertEqual(return_value, True)
119
120	def test_partially_effective_patch(self):
121	first_test_results = JSCTestResults(True, ['failure2.js'])
122	clean_test_results = JSCTestResults(False, ['failure1.js', 'failure2.js'])
123	task = self._create_task(first_test_results, clean_test_results)
124
125	return_value = task._test_patch()
126	self.assertEqual(task.test_run_count(), 2)
127	self.assertEqual(return_value, True)
128
129	def test_different_test_failures(self):
130	first_test_results = JSCTestResults(False, [])
131	clean_test_results = JSCTestResults(True, ['failure1.js', 'failure2.js'])
132	task = self._create_task(first_test_results, clean_test_results)
133
134	with self.assertRaises(ScriptError):
135	return_value = task._test_patch()
136	self.assertEqual(task.test_run_count(), 2)

Line 0 a/Tools/Scripts/webkitpy/tool/bot/jsctestresultsreader.py_sec1
- a/Tools/Scripts/webkitpy/tool/bot/jsctestresultsreader.py +47 lines
	1	# Copyright (C) 2016 Apple Inc. All rights reserved.
	2	#
	3	# Redistribution and use in source and binary forms, with or without
	4	# modification, are permitted provided that the following conditions
	5	# are met:
	6	# 1. Redistributions of source code must retain the above copyright
	7	# notice, this list of conditions and the following disclaimer.
	8	# 2. Redistributions in binary form must reproduce the above copyright
	9	# notice, this list of conditions and the following disclaimer in the
	10	# documentation and/or other materials provided with the distribution.
	11	#
	12	# THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS'' AND
	13	# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
	14	# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
	15	# DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS BE LIABLE FOR
	16	# ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
	17	# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
	18	# SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
	19	# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
	20	# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
	21	# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
	22
	23	import logging
	24
	25	from webkitpy.common.net.jsctestresults import JSCTestResults
	26
	27	_log = logging.getLogger(__name__)
	28
	29
	30	class JSCTestResultsReader(object):
	31	def __init__(self, host, results_directory):
	32	self._host = host
	33	self._results_directory = results_directory
	34
	35	def _read_file_contents(self, path):
	36	try:
	37	return self._host.filesystem.read_text_file(path)
	38	except IOError, e: # File does not exist or can't be read.
	39	return None
	40
	41	def _create_jsc_test_results(self):
	42	results_path = self._host.filesystem.join(self._results_directory, "jsc_test_results.json")
	43	contents = self._read_file_contents(results_path)
	44	return JSCTestResults.results_from_string(contents)
	45
	46	def results(self):
	47	return self._create_jsc_test_results()

Lines 1-4 a/Tools/Scripts/webkitpy/tool/bot/patchanalysistask.py_sec1
- a/Tools/Scripts/webkitpy/tool/bot/patchanalysistask.py -64 / +106 lines
1	# Copyright (c) 2010 Google Inc. All rights reserved.	1	# Copyright (c) 2010 Google Inc. All rights reserved.
		2	# Copyright (c) 2016 Apple Inc. All rights reserved.
2	#	3	#
3	# Redistribution and use in source and binary forms, with or without	4	# Redistribution and use in source and binary forms, with or without
4	# modification, are permitted provided that the following conditions are	5	# modification, are permitted provided that the following conditions are
Lines 123-149 class PatchAnalysisTask(object): a/Tools/Scripts/webkitpy/tool/bot/patchanalysistask.py_sec2
123	"Patch does not apply")	124	"Patch does not apply")
124		125
125	def _build(self):	126	def _build(self):
126	return self._run_command([	127	args = [
127	"build",	128	"build",
128	"--no-clean",	129	"--no-clean",
129	"--no-update",	130	"--no-update",
130	"--build-style=%s" % self._delegate.build_style(),	131	"--build-style=%s" % self._delegate.build_style(),
131	],	132	]
132	"Built patch",	133
133	"Patch does not build")	134	if hasattr(self._delegate, 'group'):
		135	args.append("--group=%s" % self._delegate.group())
		136
		137	return self._run_command(
		138	args,
		139	"Built patch",
		140	"Patch does not build")
134		141
135	def _build_without_patch(self):	142	def _build_without_patch(self):
136	return self._run_command([	143	args = [
137	"build",	144	"build",
138	"--force-clean",	145	"--force-clean",
139	"--no-update",	146	"--no-update",
140	"--build-style=%s" % self._delegate.build_style(),	147	"--build-style=%s" % self._delegate.build_style(),
141	],	148	]
142	"Able to build without patch",	149
143	"Unable to build without patch")	150	if hasattr(self._delegate, 'group'):
		151	args.append("--group=%s" % self._delegate.group())
		152
		153	return self._run_command(
		154	args,
		155	"Able to build without patch",
		156	"Unable to build without patch")
144		157
145	def _test(self):	158	def _test(self):
146	return self._run_command([	159	args = [
147	"build-and-test",	160	"build-and-test",
148	"--no-clean",	161	"--no-clean",
149	"--no-update",	162	"--no-update",
Lines 151-162 class PatchAnalysisTask(object): a/Tools/Scripts/webkitpy/tool/bot/patchanalysistask.py_sec3
151	"--test",	164	"--test",
152	"--non-interactive",	165	"--non-interactive",
153	"--build-style=%s" % self._delegate.build_style(),	166	"--build-style=%s" % self._delegate.build_style(),
154	],	167	]
155	"Passed tests",	168
156	"Patch does not pass tests")	169	if hasattr(self._delegate, 'group'):
		170	args.append("--group=%s" % self._delegate.group())
		171
		172	return self._run_command(
		173	args,
		174	"Passed tests",
		175	"Patch does not pass tests")
157		176
158	def _build_and_test_without_patch(self):	177	def _build_and_test_without_patch(self):
159	return self._run_command([	178	args = [
160	"build-and-test",	179	"build-and-test",
161	"--force-clean",	180	"--force-clean",
162	"--no-update",	181	"--no-update",
Lines 164-172 class PatchAnalysisTask(object): a/Tools/Scripts/webkitpy/tool/bot/patchanalysistask.py_sec4
164	"--test",	183	"--test",
165	"--non-interactive",	184	"--non-interactive",
166	"--build-style=%s" % self._delegate.build_style(),	185	"--build-style=%s" % self._delegate.build_style(),
167	],	186	]
168	"Able to pass tests without patch",	187
169	"Unable to pass tests without patch (tree is red?)")	188	if hasattr(self._delegate, 'group'):
		189	args.append("--group=%s" % self._delegate.group())
		190
		191	return self._run_command(
		192	args,
		193	"Able to pass tests without patch",
		194	"Unable to pass tests without patch (tree is red?)")
170		195
171	def _land(self):	196	def _land(self):
172	# Unclear if this should pass --quiet or not. If --parent-command always does the reporting, then it should.	197	# Unclear if this should pass --quiet or not. If --parent-command always does the reporting, then it should.
Lines 210-278 class PatchAnalysisTask(object): a/Tools/Scripts/webkitpy/tool/bot/patchanalysistask.py_sec5
210	if self._test():	235	if self._test():
211	return True	236	return True
212		237
213	# Note: archive_last_test_results deletes the results directory, making these calls order-sensitve.	238	# FIXME: Abstract out the retry logic instead of casing on the test suite here.
214	# We could remove this dependency by building the test_results from the archive.	239	if hasattr(self._delegate, 'group') and self._delegate.group() == "jsc":
215	first_results = self._delegate.test_results()	240	first_results = self._delegate.test_results()
216	first_results_archive = self._delegate.archive_last_test_results(self._patch)	241	first_script_error = self._script_error
217	first_script_error = self._script_error	242	if first_results.all_passed():
218	first_failure_status_id = self.failure_status_id	243	return True
219
220	if self._test() and not first_results.did_exceed_test_failure_limit():
221	# Only report flaky tests if we were successful at parsing results.json and archiving results.
222	if first_results and first_results_archive:
223	self._report_flaky_tests(first_results.failing_test_results(), first_results_archive)
224	return True
225
226	second_results = self._delegate.test_results()
227	second_results_archive = self._delegate.archive_last_test_results(self._patch)
228	second_script_error = self._script_error
229	second_failure_status_id = self.failure_status_id
230		244
231	if second_results.did_exceed_test_failure_limit() and first_results.did_exceed_test_failure_limit():
232	self._build_and_test_without_patch()	245	self._build_and_test_without_patch()
233	clean_tree_results = self._delegate.test_results()	246	clean_tree_results = self._delegate.test_results()
234		247
235	if (len(first_results.failing_tests()) - len(clean_tree_results.failing_tests())) <= 5:	248	# If the patch's failures are a subset of the tree's failures, the patch should pass EWS.
236	return False	249	if first_results <= clean_tree_results:
		250	return True
237		251
238	self.failure_status_id = first_failure_status_id	252	return self.report_failure(None, first_results, first_script_error)
239		253
240	return self.report_failure(first_results_archive, first_results, first_script_error)	254	else:
		255	# Note: archive_last_test_results deletes the results directory, making these calls order-sensitve.
		256	# We could remove this dependency by building the test_results from the archive.
		257	first_results = self._delegate.test_results()
		258	first_results_archive = self._delegate.archive_last_test_results(self._patch)
		259	first_script_error = self._script_error
		260	first_failure_status_id = self.failure_status_id
241		261
242	if second_results.did_exceed_test_failure_limit():	262	if self._test() and not first_results.did_exceed_test_failure_limit():
243	self._should_defer_patch_or_throw(first_results.failing_test_results(), first_results_archive, first_script_error, first_failure_status_id)	263	# Only report flaky tests if we were successful at parsing results.json and archiving results.
244	return False	264	if first_results and first_results_archive:
		265	self._report_flaky_tests(first_results.failing_test_results(), first_results_archive)
		266	return True
245		267
246	if first_results.did_exceed_test_failure_limit():	268	second_results = self._delegate.test_results()
247	self._should_defer_patch_or_throw(second_results.failing_test_results(), second_results_archive, second_script_error, second_failure_status_id)	269	second_results_archive = self._delegate.archive_last_test_results(self._patch)
248	return False	270	second_script_error = self._script_error
		271	second_failure_status_id = self.failure_status_id
249		272
250	if self._results_failed_different_tests(first_results, second_results):	273	if second_results.did_exceed_test_failure_limit() and first_results.did_exceed_test_failure_limit():
251	first_failing_results_set = frozenset(first_results.failing_test_results())	274	self._build_and_test_without_patch()
252	second_failing_results_set = frozenset(second_results.failing_test_results())	275	clean_tree_results = self._delegate.test_results()
253		276
254	tests_that_only_failed_first = first_failing_results_set.difference(second_failing_results_set)	277	if (len(first_results.failing_tests()) - len(clean_tree_results.failing_tests())) <= 5:
255	self._report_flaky_tests(tests_that_only_failed_first, first_results_archive)	278	return False
256		279
257	tests_that_only_failed_second = second_failing_results_set.difference(first_failing_results_set)	280	self.failure_status_id = first_failure_status_id
258	self._report_flaky_tests(tests_that_only_failed_second, second_results_archive)
259		281
260	tests_that_consistently_failed = first_failing_results_set.intersection(second_failing_results_set)	282	return self.report_failure(first_results_archive, first_results, first_script_error)
261	if tests_that_consistently_failed:
262	if self._should_defer_patch_or_throw(tests_that_consistently_failed, first_results_archive, first_script_error, first_failure_status_id):
263	return False # Defer patch
264		283
265	# At this point we know that at least one test flaked, but no consistent failures	284	if second_results.did_exceed_test_failure_limit():
266	# were introduced. This is a bit of a grey-zone.	285	self._should_defer_patch_or_throw(first_results.failing_test_results(), first_results_archive, first_script_error, first_failure_status_id)
267	return False # Defer patch	286	return False
268		287
269	if self._should_defer_patch_or_throw(first_results.failing_test_results(), first_results_archive, first_script_error, first_failure_status_id):	288	if first_results.did_exceed_test_failure_limit():
270	return False # Defer patch	289	self._should_defer_patch_or_throw(second_results.failing_test_results(), second_results_archive, second_script_error, second_failure_status_id)
		290	return False
271		291
272	# At this point, we know that the first and second runs had the exact same failures,	292	if self._results_failed_different_tests(first_results, second_results):
273	# and that those failures are all present on the clean tree, so we can say with certainty	293	first_failing_results_set = frozenset(first_results.failing_test_results())
274	# that the patch is good.	294	second_failing_results_set = frozenset(second_results.failing_test_results())
275	return True	295
		296	tests_that_only_failed_first = first_failing_results_set.difference(second_failing_results_set)
		297	self._report_flaky_tests(tests_that_only_failed_first, first_results_archive)
		298
		299	tests_that_only_failed_second = second_failing_results_set.difference(first_failing_results_set)
		300	self._report_flaky_tests(tests_that_only_failed_second, second_results_archive)
		301
		302	tests_that_consistently_failed = first_failing_results_set.intersection(second_failing_results_set)
		303	if tests_that_consistently_failed:
		304	if self._should_defer_patch_or_throw(tests_that_consistently_failed, first_results_archive, first_script_error, first_failure_status_id):
		305	return False # Defer patch
		306
		307	# At this point we know that at least one test flaked, but no consistent failures
		308	# were introduced. This is a bit of a grey-zone.
		309	return False # Defer patch
		310
		311	if self._should_defer_patch_or_throw(first_results.failing_test_results(), first_results_archive, first_script_error, first_failure_status_id):
		312	return False # Defer patch
		313
		314	# At this point, we know that the first and second runs had the exact same failures,
		315	# and that those failures are all present on the clean tree, so we can say with certainty
		316	# that the patch is good.
		317	return True
276		318
277	def results_archive_from_patch_test_run(self, patch):	319	def results_archive_from_patch_test_run(self, patch):
278	assert(self._patch.id() == patch.id()) # PatchAnalysisTask is not currently re-useable.	320	assert(self._patch.id() == patch.id()) # PatchAnalysisTask is not currently re-useable.

Lines 1-4 a/Tools/Scripts/webkitpy/tool/steps/build.py_sec1
- a/Tools/Scripts/webkitpy/tool/steps/build.py -7 / +17 lines
1	# Copyright (C) 2010 Google Inc. All rights reserved.	1	# Copyright (C) 2010 Google Inc. All rights reserved.
		2	# Copyright (C) 2016 Apple Inc. All rights reserved.
2	#	3	#
3	# Redistribution and use in source and binary forms, with or without	4	# Redistribution and use in source and binary forms, with or without
4	# modification, are permitted provided that the following conditions are	5	# modification, are permitted provided that the following conditions are
Lines 41-68 class Build(AbstractStep): a/Tools/Scripts/webkitpy/tool/steps/build.py_sec2
41	Options.build,	42	Options.build,
42	Options.quiet,	43	Options.quiet,
43	Options.build_style,	44	Options.build_style,
		45	Options.group,
44	]	46	]
45		47
46	def build(self, build_style):	48	def build(self, build_style, group):
47	environment = self._tool.copy_current_environment()	49	environment = self._tool.copy_current_environment()
48	environment.disable_gcc_smartquotes()	50	environment.disable_gcc_smartquotes()
49	environment.disable_jhbuild_VT100_output()	51	environment.disable_jhbuild_VT100_output()
50	env = environment.to_dictionary()	52	env = environment.to_dictionary()
51		53
52	build_webkit_command = self._tool.deprecated_port().build_webkit_command(build_style=build_style)	54	if group == "jsc":
		55	build_command = self._tool.deprecated_port().build_jsc_command(build_style=build_style)
		56	else:
		57	build_command = self._tool.deprecated_port().build_webkit_command(build_style=build_style)
53		58
54	if self._options.architecture:	59	if self._options.architecture:
55	build_webkit_command += ['ARCHS=%s' % self._options.architecture]	60	build_command += ['ARCHS=%s' % self._options.architecture]
56		61
57	self._tool.executive.run_and_throw_if_fail(build_webkit_command, self._options.quiet,	62	self._tool.executive.run_and_throw_if_fail(build_command, self._options.quiet,
58	cwd=self._tool.scm().checkout_root, env=env)	63	cwd=self._tool.scm().checkout_root, env=env)
59		64
60	def run(self, state):	65	def run(self, state):
61	if not self._options.build:	66	if not self._options.build:
62	return	67	return
63	_log.info("Building WebKit")	68	_log.info("Building WebKit")
		69
		70	group = None
		71	if hasattr(self._options, 'group'):
		72	group = self._options.group
		73
64	if self._options.build_style == "both":	74	if self._options.build_style == "both":
65	self.build("debug")	75	self.build("debug", group)
66	self.build("release")	76	self.build("release", group)
67	else:	77	else:
68	self.build(self._options.build_style)	78	self.build(self._options.build_style, group)

Lines 1-4 a/Tools/Scripts/webkitpy/tool/steps/options.py_sec1
- a/Tools/Scripts/webkitpy/tool/steps/options.py +2 lines
1	# Copyright (C) 2010 Google Inc. All rights reserved.	1	# Copyright (C) 2010 Google Inc. All rights reserved.
		2	# Copyright (C) 2016 Apple Inc. All rights reserved.
2	#	3	#
3	# Redistribution and use in source and binary forms, with or without	4	# Redistribution and use in source and binary forms, with or without
4	# modification, are permitted provided that the following conditions are	5	# modification, are permitted provided that the following conditions are
Lines 46-51 class Options(object): a/Tools/Scripts/webkitpy/tool/steps/options.py_sec2
46	email = make_option("--email", action="store", type="string", dest="email", help="Email address to use in ChangeLogs.")	47	email = make_option("--email", action="store", type="string", dest="email", help="Email address to use in ChangeLogs.")
47	force_clean = make_option("--force-clean", action="store_true", dest="force_clean", default=False, help="Clean working directory before applying patches (removes local changes and commits)")	48	force_clean = make_option("--force-clean", action="store_true", dest="force_clean", default=False, help="Clean working directory before applying patches (removes local changes and commits)")
48	git_commit = make_option("-g", "--git-commit", action="store", dest="git_commit", help="Operate on a local commit. If a range, the commits are squashed into one. <ref>.... includes the working copy changes. UPSTREAM can be used for the upstream/tracking branch.")	49	git_commit = make_option("-g", "--git-commit", action="store", dest="git_commit", help="Operate on a local commit. If a range, the commits are squashed into one. <ref>.... includes the working copy changes. UPSTREAM can be used for the upstream/tracking branch.")
		50	group = make_option("--group", action="store", dest="group", default=None, help="")
49	local_commit = make_option("--local-commit", action="store_true", dest="local_commit", default=False, help="Make a local commit for each applied patch")	51	local_commit = make_option("--local-commit", action="store_true", dest="local_commit", default=False, help="Make a local commit for each applied patch")
50	non_interactive = make_option("--non-interactive", action="store_true", dest="non_interactive", default=False, help="Never prompt the user, fail as fast as possible.")	52	non_interactive = make_option("--non-interactive", action="store_true", dest="non_interactive", default=False, help="Never prompt the user, fail as fast as possible.")
51	obsolete_patches = make_option("--no-obsolete", action="store_false", dest="obsolete_patches", default=True, help="Do not obsolete old patches before posting this one.")	53	obsolete_patches = make_option("--no-obsolete", action="store_false", dest="obsolete_patches", default=True, help="Do not obsolete old patches before posting this one.")

Lines 1-4 a/Tools/Scripts/webkitpy/tool/steps/steps_unittest.py_sec1
- a/Tools/Scripts/webkitpy/tool/steps/steps_unittest.py -2 / +5 lines
1	# Copyright (C) 2010 Google Inc. All rights reserved.	1	# Copyright (C) 2010 Google Inc. All rights reserved.
		2	# Copyright (C) 2016 Apple Inc. All rights reserved.
2	#	3	#
3	# Redistribution and use in source and binary forms, with or without	4	# Redistribution and use in source and binary forms, with or without
4	# modification, are permitted provided that the following conditions are	5	# modification, are permitted provided that the following conditions are
Lines 98-103 class StepsTest(unittest.TestCase): a/Tools/Scripts/webkitpy/tool/steps/steps_unittest.py_sec2
98	self._assert_step_output_with_bug(steps.EnsureBugIsOpenAndAssigned, 50002, expected_logs)	99	self._assert_step_output_with_bug(steps.EnsureBugIsOpenAndAssigned, 50002, expected_logs)
99		100
100	def test_runtests_args(self):	101	def test_runtests_args(self):
		102	self.maxDiff = None
101	mock_options = self._step_options()	103	mock_options = self._step_options()
102	mock_options.non_interactive = False	104	mock_options.non_interactive = False
103	mock_options.build_style = "release"	105	mock_options.build_style = "release"
Lines 111-117 MOCK run_and_throw_if_fail: ['Tools/Scripts/test-webkitpy'], cwd=/mock-checkout a/Tools/Scripts/webkitpy/tool/steps/steps_unittest.py_sec3
111	Running Perl unit tests	113	Running Perl unit tests
112	MOCK run_and_throw_if_fail: ['Tools/Scripts/test-webkitperl'], cwd=/mock-checkout	114	MOCK run_and_throw_if_fail: ['Tools/Scripts/test-webkitperl'], cwd=/mock-checkout
113	Running JavaScriptCore tests	115	Running JavaScriptCore tests
114	MOCK run_and_throw_if_fail: ['Tools/Scripts/run-javascriptcore-tests'], cwd=/mock-checkout	116	MOCK run_and_throw_if_fail: ['Tools/Scripts/run-javascriptcore-tests', '--no-fail-fast', '--json-output=jsc_test_results.json'], cwd=/mock-checkout
115	Running bindings generation tests	117	Running bindings generation tests
116	MOCK run_and_throw_if_fail: ['Tools/Scripts/run-bindings-tests'], cwd=/mock-checkout	118	MOCK run_and_throw_if_fail: ['Tools/Scripts/run-bindings-tests'], cwd=/mock-checkout
117	Running run-webkit-tests	119	Running run-webkit-tests
Lines 120-125 MOCK run_and_throw_if_fail: ['Tools/Scripts/run-webkit-tests', '--release', '--q a/Tools/Scripts/webkitpy/tool/steps/steps_unittest.py_sec4
120	OutputCapture().assert_outputs(self, step.run, [{}], expected_logs=expected_logs)	122	OutputCapture().assert_outputs(self, step.run, [{}], expected_logs=expected_logs)
121		123
122	def test_runtests_debug_args(self):	124	def test_runtests_debug_args(self):
		125	self.maxDiff = None
123	mock_options = self._step_options()	126	mock_options = self._step_options()
124	mock_options.non_interactive = False	127	mock_options.non_interactive = False
125	mock_options.build_style = "debug"	128	mock_options.build_style = "debug"
Lines 133-139 MOCK run_and_throw_if_fail: ['Tools/Scripts/test-webkitpy'], cwd=/mock-checkout a/Tools/Scripts/webkitpy/tool/steps/steps_unittest.py_sec5
133	Running Perl unit tests	136	Running Perl unit tests
134	MOCK run_and_throw_if_fail: ['Tools/Scripts/test-webkitperl'], cwd=/mock-checkout	137	MOCK run_and_throw_if_fail: ['Tools/Scripts/test-webkitperl'], cwd=/mock-checkout
135	Running JavaScriptCore tests	138	Running JavaScriptCore tests
136	MOCK run_and_throw_if_fail: ['Tools/Scripts/run-javascriptcore-tests'], cwd=/mock-checkout	139	MOCK run_and_throw_if_fail: ['Tools/Scripts/run-javascriptcore-tests', '--no-fail-fast', '--json-output=jsc_test_results.json'], cwd=/mock-checkout
137	Running bindings generation tests	140	Running bindings generation tests
138	MOCK run_and_throw_if_fail: ['Tools/Scripts/run-bindings-tests'], cwd=/mock-checkout	141	MOCK run_and_throw_if_fail: ['Tools/Scripts/run-bindings-tests'], cwd=/mock-checkout
139	Running run-webkit-tests	142	Running run-webkit-tests