Attachment #297551 for bug #162458

View Details Raw Unified Return to Bug 162458

| Differences between



2016-12-20  Srinivasan Vijayaraghavan  <svijayaraghavan@apple.com>

        EWS should run JavaScriptCore tests
        https://bugs.webkit.org/show_bug.cgi?id=162458

        Reviewed by NOBODY (OOPS!).

        * Scripts/webkitpy/common/config/ews.json: Add config info for JSC EWS.
        * Scripts/webkitpy/common/config/ports.py: Add support for JSC EWS in Mac Port.
        (DeprecatedPort.build_jsc_command): Added - command to build JSC only (quicker than building all of WebKit).
        (DeprecatedPort.run_javascriptcore_tests_command): Allow JSC EWS to only run JSC tests.
        (MacPort.run_webkit_tests_command): Check for JSC.
        * Scripts/webkitpy/common/net/jsctestresults.py: Added.
        (JSCTestResults):
        (JSCTestResults.results_from_string): Use json library to parse results.
        (JSCTestResults.__init__): Initialize members.
        (JSCTestResults.__eq__): This enables unit testing.
        (JSCTestResults.stress_test_failures): Getter.
        (JSCTestResults.all_api_tests_passed): Getter.
        (JSCTestResults.all_passed): Getter.
        (JSCTestResults.failing_test_results): Getter. Some of these exist for interopability reasons.
        (JSCTestResults.test_results): Getter.
        (JSCTestResults.failing_tests): Getter.
        (JSCTestResults.did_exceed_test_failure_limit): Getter.
        * Scripts/webkitpy/common/net/jsctestresults_unittest.py: Added.
        (JSCTestResultsTest): Class to test JSCTestResults.
        (JSCTestResultsTest.test_results_from_string): Tests parsing.
        * Scripts/webkitpy/tool/bot/jsctestresultsreader.py: Added.
        (JSCTestResultsReader): Reads results file.
        (JSCTestResultsReader.__init__): Sets attributes.
        (JSCTestResultsReader._read_file_contents): Reads file.
        (JSCTestResultsReader._create_jsc_test_results): Creates a JSCTestResults object.
        (JSCTestResultsReader.results): Invokes _create_jsc_test_results to return a JSCTestResults object.
        * Scripts/webkitpy/tool/bot/patchanalysistask.py:
        (PatchAnalysisTask._build): Check for JSC.
        (PatchAnalysisTask._build_without_patch): Check for JSC.
        (PatchAnalysisTask._test): Check for JSC.
        (PatchAnalysisTask._build_and_test_without_patch): Check for JSC.
        (PatchAnalysisTask._test_patch): Add retry logic for JSC.
        * Scripts/webkitpy/tool/commands/earlywarningsystem.py:
        (AbstractEarlyWarningSystem.begin_work_queue): Use JSCTestResultsReader instead of LayoutTestResultsReader in JSC EWS.
        (AbstractEarlyWarningSystem.test_results): _layout_test_results_reader -> _test_results_reader.
        (AbstractEarlyWarningSystem.archive_last_test_results): _layout_test_results_reader -> _test_results_reader.
        (AbstractEarlyWarningSystem.group): This attribute determines the type of EWS (eg. JSC).
        (AbstractEarlyWarningSystem.load_ews_classes): Add _group.
        * Scripts/webkitpy/tool/commands/earlywarningsystem_unittest.py:
        (EarlyWarningSystemTest._default_expected_logs): Add _group.
        * Scripts/webkitpy/tool/steps/build.py:
        (Build.options): Check for JSC.
        (Build.build): Check for JSC.
        (Build.run): Check for JSC.
        * Scripts/webkitpy/tool/steps/options.py:
        (Options): Add --group command line option.
        * Scripts/webkitpy/tool/steps/runtests.py:
        (RunTests.options): Add group.
        (RunTests.run): Check for JSC.
        (RunTests._run_webkit_tests): Check for JSC.
        (RunTests): Add _group attribute.
        (RunTests._run_javascriptcore_tests): New.
        * Scripts/webkitpy/tool/steps/steps_unittest.py: Change expected command for running jsc tests.

2016-09-25  Gyuyoung Kim  <gyuyoung.kim@webkit.org>

        [EFL] Adjust EFL indentation to EFL MiniBrowser

Lines 50-54 a/Tools/Scripts/webkitpy/common/config/ews.json_sec1
- a/Tools/Scripts/webkitpy/common/config/ews.json +6 lines
50	"port": "mac",	50	"port": "mac",
51	"name": "mac-32bit-ews",	51	"name": "mac-32bit-ews",
52	"architecture": "i386"	52	"architecture": "i386"
		53	},
		54	"JSC EWS": {
		55	"port": "mac",
		56	"name": "jsc-ews",
		57	"group": "jsc",
		58	"runTests": true
53	}	59	}
54	}	60	}



# Copyright (C) 2009, Google Inc. All rights reserved.
# Copyright (C) 2013 Nokia Corporation and/or its subsidiary(-ies).
# Copyright (C) 2015-2016 Apple Inc. All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are

            command.append("--release")
        return command

    def build_jsc_command(self, build_style=None):
        command = self.script_shell_command("build-jsc")
        if build_style == "debug":
            command.append("--debug")
        return command

    def run_javascriptcore_tests_command(self, build_style=None):
        command = self.script_shell_command("run-javascriptcore-tests")
        command.append("--no-fail-fast")
        command.append("--json-output=jsc_test_results.json")
        if (build_style == "debug"):
            command.append("--debug")
        return command

    def run_webkit_tests_command(self, build_style=None):
        command = self.script_shell_command("run-webkit-tests")

class MacPort(DeprecatedPort):
    port_flag_name = "mac"

    def run_webkit_tests_command(self, build_style=None, test_suite=None):
        if (test_suite == "jsc"):
            return super(MacPort, self).run_javascriptcore_tests_command(build_style)

        command = super(MacPort, self).run_webkit_tests_command(build_style)
        command.append("--dump-render-tree")
        return command



# Copyright (C) 2016 Apple Inc. All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions
# are met:
# 1.  Redistributions of source code must retain the above copyright
#     notice, this list of conditions and the following disclaimer.
# 2.  Redistributions in binary form must reproduce the above copyright
#     notice, this list of conditions and the following disclaimer in the
#     documentation and/or other materials provided with the distribution.
#
# THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS'' AND
# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
# DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS BE LIABLE FOR
# ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
# SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

import json
import logging

_log = logging.getLogger(__name__)


class JSCTestResults(object):
    @classmethod
    def results_from_string(cls, string):
        if not string:
            return None
        parsed_results = json.loads(string)

        return cls(parsed_results['allApiTestsPassed'], parsed_results['stressTestFailures'])

    def __init__(self, all_api_tests_passed, stress_test_failures):
        self._all_api_tests_passed = all_api_tests_passed
        self._stress_test_failures = stress_test_failures

        self._test_results = stress_test_failures
        if self._all_api_tests_passed == False:
            self._test_results += "apiTests"

    def __eq__(self, other):
        return self._all_api_tests_passed == other._all_api_tests_passed and set(self._stress_test_failures) == set(other._stress_test_failures)

    def stress_test_failures(self):
        return self._stress_test_failures

    def all_api_tests_passed(self):
        return self._all_api_tests_passed

    def all_passed(self):
        return self._all_api_tests_passed == True and len(self._stress_test_failures) == 0

    def failing_test_results(self):
        return self.stress_test_failures()

    def test_results(self):
        return self._test_results

    def failing_tests(self):
        return self._test_results

    # No defined failure limit for JSC
    def did_exceed_test_failure_limit(self):
        return False



# Copyright (C) 2016 Apple Inc. All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions
# are met:
# 1.  Redistributions of source code must retain the above copyright
#     notice, this list of conditions and the following disclaimer.
# 2.  Redistributions in binary form must reproduce the above copyright
#     notice, this list of conditions and the following disclaimer in the
#     documentation and/or other materials provided with the distribution.
#
# THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS'' AND
# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
# DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS BE LIABLE FOR
# ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
# SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

import unittest

from webkitpy.common.net.jsctestresults import JSCTestResults


class JSCTestResultsTest(unittest.TestCase):
    def test_results_from_string(self):
        no_failures_string = '{"allApiTestsPassed": true, "stressTestFailures":[]}'
        no_failures_results = JSCTestResults(True, [])
        self.assertEqual(no_failures_results, JSCTestResults.results_from_string(no_failures_string))

        api_test_failures_string = '{"allApiTestsPassed": false, "stressTestFailures":[]}'
        api_test_failures_results = JSCTestResults(False, [])
        self.assertEqual(api_test_failures_results, JSCTestResults.results_from_string(api_test_failures_string))

        many_failures_string = '{"allApiTestsPassed": false, "stressTestFailures":["es6.yaml/es6/typed_arrays_Int16Array.js.default", "es6.yaml/es6/typed_arrays_Int8Array.js.default"]}'
        many_failures_results = JSCTestResults(False, ["es6.yaml/es6/typed_arrays_Int16Array.js.default", "es6.yaml/es6/typed_arrays_Int8Array.js.default"])
        self.assertEqual(many_failures_results, JSCTestResults.results_from_string(many_failures_string))



# Copyright (C) 2016 Apple Inc. All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions
# are met:
# 1.  Redistributions of source code must retain the above copyright
#     notice, this list of conditions and the following disclaimer.
# 2.  Redistributions in binary form must reproduce the above copyright
#     notice, this list of conditions and the following disclaimer in the
#     documentation and/or other materials provided with the distribution.
#
# THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS'' AND
# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
# DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS BE LIABLE FOR
# ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
# SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

import logging
import json
from webkitpy.common.net.jsctestresults import JSCTestResults

_log = logging.getLogger(__name__)


class JSCTestResultsReader(object):
    def __init__(self, host, results_directory):
        self._host = host
        self._results_directory = results_directory

    def _read_file_contents(self, path):
        try:
            return self._host.filesystem.read_text_file(path)
        except IOError, e:  # File does not exist or can't be read.
            return None

    def _create_jsc_test_results(self):
        results_path = self._host.filesystem.join(self._results_directory, "jsc_test_results.json")
        contents = self._read_file_contents(results_path)
        return JSCTestResults.results_from_string(contents)

    def results(self):
        return self._create_jsc_test_results()



# Copyright (c) 2010 Google Inc. All rights reserved.
# Copyright (c) 2016 Apple Inc. All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are

        "Patch does not apply")

    def _build(self):
        args = [
            "build",
            "--no-clean",
            "--no-update",
            "--build-style=%s" % self._delegate.build_style(),
        ]

        try:
            args.append("--group=%s" % self._delegate.group())
        finally:
            return self._run_command(
                args,
                "Built patch",
                "Patch does not build")

    def _build_without_patch(self):
        args = [
            "build",
            "--force-clean",
            "--no-update",
            "--build-style=%s" % self._delegate.build_style(),
        ]

        try:
            args.append("--group=%s" % self._delegate.group())
        finally:
            return self._run_command(
                args,
                "Able to build without patch",
                "Unable to build without patch")

    def _test(self):
        args = [
            "build-and-test",
            "--no-clean",
            "--no-update",

            "--test",
            "--non-interactive",
            "--build-style=%s" % self._delegate.build_style(),
        ]

        try:
            args.append("--group=%s" % self._delegate.group())
        finally:
            return self._run_command(
                args,
                "Passed tests",
                "Patch does not pass tests")

    def _build_and_test_without_patch(self):
        args = [
            "build-and-test",
            "--force-clean",
            "--no-update",

            "--test",
            "--non-interactive",
            "--build-style=%s" % self._delegate.build_style(),
        ]

        try:
            args.append("--group=%s" % self._delegate.group())
        finally:
            return self._run_command(
                args,
                "Able to pass tests without patch",
                "Unable to pass tests without patch (tree is red?)")

    def _land(self):
        # Unclear if this should pass --quiet or not.  If --parent-command always does the reporting, then it should.

        if self._test():
            return True

        if hasattr(self._delegate, 'group') and self._delegate.group() == "jsc":
            first_results = self._delegate.test_results()
            first_script_error = self._script_error
            if first_results.all_api_tests_passed() is True and len(first_results.stress_test_failures()) == 0:
                return True
            else:
                # TODO svijayaraghavan@apple.com: Comment this, and uncomment the below lines so it actually builds without patch
                #clean_tree_results = first_results
                #clean_tree_results['allApiTestsPassed'] = False
            if first_results and first_results_archive:
                self._report_flaky_tests(first_results.failing_test_results(), first_results_archive)
            return True

                self._build_and_test_without_patch()
                clean_tree_results = self._delegate.test_results()
        second_script_error = self._script_error
        second_failure_status_id = self.failure_status_id

                if clean_tree_results == first_results:
                    return True
            clean_tree_results = self._delegate.test_results()

                return self.report_failure(None, first_results, first_script_error)
                return False

        else:
            # Note: archive_last_test_results deletes the results directory, making these calls order-sensitve.
            # We could remove this dependency by building the test_results from the archive.
            first_results = self._delegate.test_results()
            first_results_archive = self._delegate.archive_last_test_results(self._patch)
            first_script_error = self._script_error
            first_failure_status_id = self.failure_status_id

            if self._test() and not first_results.did_exceed_test_failure_limit():
                # Only report flaky tests if we were successful at parsing results.json and archiving results.
                if first_results and first_results_archive:
                    self._report_flaky_tests(first_results.failing_test_results(), first_results_archive)
                return True

            second_results = self._delegate.test_results()
            second_results_archive = self._delegate.archive_last_test_results(self._patch)
            second_script_error = self._script_error
            second_failure_status_id = self.failure_status_id

            if second_results.did_exceed_test_failure_limit() and first_results.did_exceed_test_failure_limit():
                self._build_and_test_without_patch()
                clean_tree_results = self._delegate.test_results()

                if (len(first_results.failing_tests()) - len(clean_tree_results.failing_tests())) <= 5:
                    return False

                self.failure_status_id = first_failure_status_id
            first_failing_results_set = frozenset(first_results.failing_test_results())
            second_failing_results_set = frozenset(second_results.failing_test_results())

                return self.report_failure(first_results_archive, first_results, first_script_error)
            self._report_flaky_tests(tests_that_only_failed_first, first_results_archive)

            if second_results.did_exceed_test_failure_limit():
                self._should_defer_patch_or_throw(first_results.failing_test_results(), first_results_archive, first_script_error, first_failure_status_id)
                return False

            if first_results.did_exceed_test_failure_limit():
                self._should_defer_patch_or_throw(second_results.failing_test_results(), second_results_archive, second_script_error, second_failure_status_id)
                return False

            if self._results_failed_different_tests(first_results, second_results):
                first_failing_results_set = frozenset(first_results.failing_test_results())
                second_failing_results_set = frozenset(second_results.failing_test_results())

                tests_that_only_failed_first = first_failing_results_set.difference(second_failing_results_set)
                self._report_flaky_tests(tests_that_only_failed_first, first_results_archive)
                if self._should_defer_patch_or_throw(tests_that_consistently_failed, first_results_archive, first_script_error, first_failure_status_id):
                    return False  # Defer patch

                tests_that_only_failed_second = second_failing_results_set.difference(first_failing_results_set)
                self._report_flaky_tests(tests_that_only_failed_second, second_results_archive)
            return False  # Defer patch

                tests_that_consistently_failed = first_failing_results_set.intersection(second_failing_results_set)
                if tests_that_consistently_failed:
                    if self._should_defer_patch_or_throw(tests_that_consistently_failed, first_results_archive, first_script_error, first_failure_status_id):
                        return False  # Defer patch

                # At this point we know that at least one test flaked, but no consistent failures
                # were introduced. This is a bit of a grey-zone.
                return False  # Defer patch

            if self._should_defer_patch_or_throw(first_results.failing_test_results(), first_results_archive, first_script_error, first_failure_status_id):
                return False  # Defer patch

            # At this point, we know that the first and second runs had the exact same failures,
            # and that those failures are all present on the clean tree, so we can say with certainty
            # that the patch is good.
            return True

    def results_archive_from_patch_test_run(self, patch):
        assert(self._patch.id() == patch.id())  # PatchAnalysisTask is not currently re-useable.

Lines 1-4 a/Tools/Scripts/webkitpy/tool/commands/earlywarningsystem.py_sec1
- a/Tools/Scripts/webkitpy/tool/commands/earlywarningsystem.py -3 / +18 lines
1	# Copyright (c) 2009 Google Inc. All rights reserved.	1	# Copyright (c) 2009 Google Inc. All rights reserved.
		2	# Copyright (c) 2016 Apple Inc. All rights reserved.
2	#	3	#
3	# Redistribution and use in source and binary forms, with or without	4	# Redistribution and use in source and binary forms, with or without
4	# modification, are permitted provided that the following conditions are	5	# modification, are permitted provided that the following conditions are
Lines 28-33 a/Tools/Scripts/webkitpy/tool/commands/earlywarningsystem.py_sec2
28		29
29	import json	30	import json
30	import logging	31	import logging
		32	import os
31	from optparse import make_option	33	from optparse import make_option
32		34
33	from webkitpy.common.config.committers import CommitterList	35	from webkitpy.common.config.committers import CommitterList
Lines 36-41 from webkitpy.common.system.filesystem import FileSystem a/Tools/Scripts/webkitpy/tool/commands/earlywarningsystem.py_sec3
36	from webkitpy.common.system.executive import ScriptError	38	from webkitpy.common.system.executive import ScriptError
37	from webkitpy.tool.bot.earlywarningsystemtask import EarlyWarningSystemTask, EarlyWarningSystemTaskDelegate	39	from webkitpy.tool.bot.earlywarningsystemtask import EarlyWarningSystemTask, EarlyWarningSystemTaskDelegate
38	from webkitpy.tool.bot.layouttestresultsreader import LayoutTestResultsReader	40	from webkitpy.tool.bot.layouttestresultsreader import LayoutTestResultsReader
		41	from webkitpy.tool.bot.jsctestresultsreader import JSCTestResultsReader
39	from webkitpy.tool.bot.patchanalysistask import UnableToApplyPatch, PatchIsNotValid	42	from webkitpy.tool.bot.patchanalysistask import UnableToApplyPatch, PatchIsNotValid
40	from webkitpy.tool.bot.queueengine import QueueEngine	43	from webkitpy.tool.bot.queueengine import QueueEngine
41	from webkitpy.tool.commands.queues import AbstractReviewQueue	44	from webkitpy.tool.commands.queues import AbstractReviewQueue
Lines 53-59 class AbstractEarlyWarningSystem(AbstractReviewQueue, EarlyWarningSystemTaskDele a/Tools/Scripts/webkitpy/tool/commands/earlywarningsystem.py_sec4
53		56
54	def begin_work_queue(self):	57	def begin_work_queue(self):
55	AbstractReviewQueue.begin_work_queue(self)	58	AbstractReviewQueue.begin_work_queue(self)
56	self._layout_test_results_reader = LayoutTestResultsReader(self._tool, self._port.results_directory(), self._log_directory())	59
		60	if self.group() == "jsc":
		61	dir = os.path.dirname(os.path.dirname(os.path.dirname(self._port.results_directory())))
		62	self._test_results_reader = JSCTestResultsReader(self._tool, dir)
		63	else:
		64	self._test_results_reader = LayoutTestResultsReader(self._tool, self._port.results_directory(), self._log_directory())
57		65
58	def _failing_tests_message(self, task, patch):	66	def _failing_tests_message(self, task, patch):
59	results = task.results_from_patch_test_run(patch)	67	results = task.results_from_patch_test_run(patch)
Lines 117-130 class AbstractEarlyWarningSystem(AbstractReviewQueue, EarlyWarningSystemTaskDele a/Tools/Scripts/webkitpy/tool/commands/earlywarningsystem.py_sec5
117	return self._update_status(message, patch=patch, results_file=failure_log)	125	return self._update_status(message, patch=patch, results_file=failure_log)
118		126
119	def test_results(self):	127	def test_results(self):
120	return self._layout_test_results_reader.results()	128	return self._test_results_reader.results()
121		129
122	def archive_last_test_results(self, patch):	130	def archive_last_test_results(self, patch):
123	return self._layout_test_results_reader.archive(patch)	131	return self._test_results_reader.archive(patch)
124		132
125	def build_style(self):	133	def build_style(self):
126	return self._build_style	134	return self._build_style
127		135
		136	def group(self):
		137	if hasattr(self, "_group"):
		138	return self._group
		139	else:
		140	return None
		141
128	def refetch_patch(self, patch):	142	def refetch_patch(self, patch):
129	return self._tool.bugs.fetch_attachment(patch.id())	143	return self._tool.bugs.fetch_attachment(patch.id())
130		144
Lines 156-160 class AbstractEarlyWarningSystem(AbstractReviewQueue, EarlyWarningSystemTaskDele a/Tools/Scripts/webkitpy/tool/commands/earlywarningsystem.py_sec6
156	'_build_style': config.get('style', "release"),	170	'_build_style': config.get('style', "release"),
157	'watchers': config.get('watchers', []),	171	'watchers': config.get('watchers', []),
158	'run_tests': config.get('runTests', cls.run_tests),	172	'run_tests': config.get('runTests', cls.run_tests),
		173	'_group': config.get('group', None),
159	}))	174	}))
160	return classes	175	return classes



# Copyright (C) 2009 Google Inc. All rights reserved.
# Copyright (C) 2016 Apple Inc. All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are

            "port": ews.port_name,
            "architecture": " --architecture=%s" % ews.architecture if ews.architecture else "",
            "build_style": ews.build_style(),
            "group": ews.group(),
        }
        if ews.run_tests:
            run_tests_line = "Running: webkit-patch --status-host=example.com build-and-test --no-clean --no-update --test --non-interactive --build-style=%(build_style)s --group=%(group)s --port=%(port)s%(architecture)s\n" % string_replacements
        else:
            run_tests_line = ""
        string_replacements['run_tests_line'] = run_tests_line

Running: webkit-patch --status-host=example.com clean --port=%(port)s%(architecture)s
Running: webkit-patch --status-host=example.com update --port=%(port)s%(architecture)s
Running: webkit-patch --status-host=example.com apply-attachment --no-update --non-interactive 10000 --port=%(port)s%(architecture)s
Running: webkit-patch --status-host=example.com build --no-clean --no-update --build-style=%(build_style)s --group=%(group)s --port=%(port)s%(architecture)s
%(run_tests_line)sMOCK: update_status: %(name)s Pass
MOCK: release_work_item: %(name)s 10000
""" % string_replacements,



# Copyright (C) 2010 Google Inc. All rights reserved.
# Copyright (C) 2016 Apple Inc. All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are

            Options.build,
            Options.quiet,
            Options.build_style,
            Options.group,
        ]

    def build(self, build_style, group):
        environment = self._tool.copy_current_environment()
        environment.disable_gcc_smartquotes()
        environment.disable_jhbuild_VT100_output()
        env = environment.to_dictionary()

        if group == "jsc":
            build_command = self._tool.deprecated_port().build_jsc_command(build_style=build_style)
        else:
            build_command = self._tool.deprecated_port().build_webkit_command(build_style=build_style)

        if self._options.architecture:
            build_command += ['ARCHS=%s' % self._options.architecture]

        self._tool.executive.run_and_throw_if_fail(build_command, self._options.quiet,
            cwd=self._tool.scm().checkout_root, env=env)

    def run(self, state):
        if not self._options.build:
            return
        _log.info("Building WebKit")

        if hasattr(self._options, 'group') and self._options.group == "jsc":
            group = self._options.group
        else:
            group = None

        if self._options.build_style == "both":
            self.build("debug", group)
            self.build("release", group)
        else:
            self.build(self._options.build_style, group)



# Copyright (C) 2010 Google Inc. All rights reserved.
# Copyright (C) 2016 Apple Inc. All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are

    email = make_option("--email", action="store", type="string", dest="email", help="Email address to use in ChangeLogs.")
    force_clean = make_option("--force-clean", action="store_true", dest="force_clean", default=False, help="Clean working directory before applying patches (removes local changes and commits)")
    git_commit = make_option("-g", "--git-commit", action="store", dest="git_commit", help="Operate on a local commit. If a range, the commits are squashed into one. <ref>.... includes the working copy changes. UPSTREAM can be used for the upstream/tracking branch.")
    group = make_option("--group", action="store", dest="group", default=None, help="")
    local_commit = make_option("--local-commit", action="store_true", dest="local_commit", default=False, help="Make a local commit for each applied patch")
    non_interactive = make_option("--non-interactive", action="store_true", dest="non_interactive", default=False, help="Never prompt the user, fail as fast as possible.")
    obsolete_patches = make_option("--no-obsolete", action="store_false", dest="obsolete_patches", default=True, help="Do not obsolete old patches before posting this one.")

Lines 1-4 a/Tools/Scripts/webkitpy/tool/steps/runtests.py_sec1
- a/Tools/Scripts/webkitpy/tool/steps/runtests.py +10 lines
1	# Copyright (C) 2010 Google Inc. All rights reserved.	1	# Copyright (C) 2010 Google Inc. All rights reserved.
		2	# Copyright (C) 2016 Apple Inc. All rights reserved.
2	#	3	#
3	# Redistribution and use in source and binary forms, with or without	4	# Redistribution and use in source and binary forms, with or without
4	# modification, are permitted provided that the following conditions are	5	# modification, are permitted provided that the following conditions are
Lines 53-58 class RunTests(AbstractStep): a/Tools/Scripts/webkitpy/tool/steps/runtests.py_sec2
53	Options.iterate_on_new_tests,	54	Options.iterate_on_new_tests,
54	Options.non_interactive,	55	Options.non_interactive,
55	Options.quiet,	56	Options.quiet,
		57	Options.group,
56	]	58	]
57		59
58	def run(self, state):	60	def run(self, state):
Lines 63-68 class RunTests(AbstractStep): a/Tools/Scripts/webkitpy/tool/steps/runtests.py_sec3
63	if not self._options.test:	65	if not self._options.test:
64	return	66	return
65		67
		68	if hasattr(self._options, "group") and self._options.group == "jsc":
		69	self._run_javascriptcore_tests()
		70	return
		71
66	python_unittests_command = self._tool.deprecated_port().run_python_unittests_command()	72	python_unittests_command = self._tool.deprecated_port().run_python_unittests_command()
67	if python_unittests_command:	73	if python_unittests_command:
68	_log.info("Running Python unit tests")	74	_log.info("Running Python unit tests")
Lines 143-145 class RunTests(AbstractStep): a/Tools/Scripts/webkitpy/tool/steps/runtests.py_sec4
143	args.append("--iterations=%d" % iterate_on_new_tests)	149	args.append("--iterations=%d" % iterate_on_new_tests)
144		150
145	self._tool.executive.run_and_throw_if_fail(args, cwd=self._tool.scm().checkout_root)	151	self._tool.executive.run_and_throw_if_fail(args, cwd=self._tool.scm().checkout_root)
		152
		153	def _run_javascriptcore_tests(self):
		154	args = self._tool.deprecated_port().run_javascriptcore_tests_command(self._options.build_style)
		155	self._tool.executive.run_and_throw_if_fail(args, cwd=self._tool.scm().checkout_root)



# Copyright (C) 2010 Google Inc. All rights reserved.
# Copyright (C) 2016 Apple Inc. All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are

        self._assert_step_output_with_bug(steps.EnsureBugIsOpenAndAssigned, 50002, expected_logs)

    def test_runtests_args(self):
        self.maxDiff = None
        mock_options = self._step_options()
        mock_options.non_interactive = False
        mock_options.build_style = "release"

Running Perl unit tests
MOCK run_and_throw_if_fail: ['Tools/Scripts/test-webkitperl'], cwd=/mock-checkout
Running JavaScriptCore tests
MOCK run_and_throw_if_fail: ['Tools/Scripts/run-javascriptcore-tests', '--no-fail-fast', '--json-output=jsc_test_results.json'], cwd=/mock-checkout
Running bindings generation tests
MOCK run_and_throw_if_fail: ['Tools/Scripts/run-bindings-tests'], cwd=/mock-checkout
Running run-webkit-tests

        OutputCapture().assert_outputs(self, step.run, [{}], expected_logs=expected_logs)

    def test_runtests_debug_args(self):
        self.maxDiff = None
        mock_options = self._step_options()
        mock_options.non_interactive = False
        mock_options.build_style = "debug"

Running Perl unit tests
MOCK run_and_throw_if_fail: ['Tools/Scripts/test-webkitperl'], cwd=/mock-checkout
Running JavaScriptCore tests
MOCK run_and_throw_if_fail: ['Tools/Scripts/run-javascriptcore-tests', '--no-fail-fast', '--json-output=jsc_test_results.json'], cwd=/mock-checkout
Running bindings generation tests
MOCK run_and_throw_if_fail: ['Tools/Scripts/run-bindings-tests'], cwd=/mock-checkout
Running run-webkit-tests

Return to Bug 162458

Lines 1-3 a/Tools/ChangeLog_sec1
- a/Tools/ChangeLog +61 lines
		1	2016-12-20 Srinivasan Vijayaraghavan <svijayaraghavan@apple.com>
		2
		3	EWS should run JavaScriptCore tests
		4	https://bugs.webkit.org/show_bug.cgi?id=162458
		5
		6	Reviewed by NOBODY (OOPS!).
		7
		8	* Scripts/webkitpy/common/config/ews.json: Add config info for JSC EWS.
		9	* Scripts/webkitpy/common/config/ports.py: Add support for JSC EWS in Mac Port.
		10	(DeprecatedPort.build_jsc_command): Added - command to build JSC only (quicker than building all of WebKit).
		11	(DeprecatedPort.run_javascriptcore_tests_command): Allow JSC EWS to only run JSC tests.
		12	(MacPort.run_webkit_tests_command): Check for JSC.
		13	* Scripts/webkitpy/common/net/jsctestresults.py: Added.
		14	(JSCTestResults):
		15	(JSCTestResults.results_from_string): Use json library to parse results.
		16	(JSCTestResults.__init__): Initialize members.
		17	(JSCTestResults.__eq__): This enables unit testing.
		18	(JSCTestResults.stress_test_failures): Getter.
		19	(JSCTestResults.all_api_tests_passed): Getter.
		20	(JSCTestResults.all_passed): Getter.
		21	(JSCTestResults.failing_test_results): Getter. Some of these exist for interopability reasons.
		22	(JSCTestResults.test_results): Getter.
		23	(JSCTestResults.failing_tests): Getter.
		24	(JSCTestResults.did_exceed_test_failure_limit): Getter.
		25	* Scripts/webkitpy/common/net/jsctestresults_unittest.py: Added.
		26	(JSCTestResultsTest): Class to test JSCTestResults.
		27	(JSCTestResultsTest.test_results_from_string): Tests parsing.
		28	* Scripts/webkitpy/tool/bot/jsctestresultsreader.py: Added.
		29	(JSCTestResultsReader): Reads results file.
		30	(JSCTestResultsReader.__init__): Sets attributes.
		31	(JSCTestResultsReader._read_file_contents): Reads file.
		32	(JSCTestResultsReader._create_jsc_test_results): Creates a JSCTestResults object.
		33	(JSCTestResultsReader.results): Invokes _create_jsc_test_results to return a JSCTestResults object.
		34	* Scripts/webkitpy/tool/bot/patchanalysistask.py:
		35	(PatchAnalysisTask._build): Check for JSC.
		36	(PatchAnalysisTask._build_without_patch): Check for JSC.
		37	(PatchAnalysisTask._test): Check for JSC.
		38	(PatchAnalysisTask._build_and_test_without_patch): Check for JSC.
		39	(PatchAnalysisTask._test_patch): Add retry logic for JSC.
		40	* Scripts/webkitpy/tool/commands/earlywarningsystem.py:
		41	(AbstractEarlyWarningSystem.begin_work_queue): Use JSCTestResultsReader instead of LayoutTestResultsReader in JSC EWS.
		42	(AbstractEarlyWarningSystem.test_results): _layout_test_results_reader -> _test_results_reader.
		43	(AbstractEarlyWarningSystem.archive_last_test_results): _layout_test_results_reader -> _test_results_reader.
		44	(AbstractEarlyWarningSystem.group): This attribute determines the type of EWS (eg. JSC).
		45	(AbstractEarlyWarningSystem.load_ews_classes): Add _group.
		46	* Scripts/webkitpy/tool/commands/earlywarningsystem_unittest.py:
		47	(EarlyWarningSystemTest._default_expected_logs): Add _group.
		48	* Scripts/webkitpy/tool/steps/build.py:
		49	(Build.options): Check for JSC.
		50	(Build.build): Check for JSC.
		51	(Build.run): Check for JSC.
		52	* Scripts/webkitpy/tool/steps/options.py:
		53	(Options): Add --group command line option.
		54	* Scripts/webkitpy/tool/steps/runtests.py:
		55	(RunTests.options): Add group.
		56	(RunTests.run): Check for JSC.
		57	(RunTests._run_webkit_tests): Check for JSC.
		58	(RunTests): Add _group attribute.
		59	(RunTests._run_javascriptcore_tests): New.
		60	* Scripts/webkitpy/tool/steps/steps_unittest.py: Change expected command for running jsc tests.
		61
1	2016-09-25 Gyuyoung Kim <gyuyoung.kim@webkit.org>	62	2016-09-25 Gyuyoung Kim <gyuyoung.kim@webkit.org>
2		63
3	[EFL] Adjust EFL indentation to EFL MiniBrowser	64	[EFL] Adjust EFL indentation to EFL MiniBrowser

Lines 1-6 a/Tools/Scripts/webkitpy/common/config/ports.py_sec1
- a/Tools/Scripts/webkitpy/common/config/ports.py -4 / +18 lines
1	# Copyright (C) 2009, Google Inc. All rights reserved.	1	# Copyright (C) 2009, Google Inc. All rights reserved.
2	# Copyright (C) 2013 Nokia Corporation and/or its subsidiary(-ies).	2	# Copyright (C) 2013 Nokia Corporation and/or its subsidiary(-ies).
3	# Copyright (C) 2015 Apple Inc. All rights reserved.	3	# Copyright (C) 2015-2016 Apple Inc. All rights reserved.
4	#	4	#
5	# Redistribution and use in source and binary forms, with or without	5	# Redistribution and use in source and binary forms, with or without
6	# modification, are permitted provided that the following conditions are	6	# modification, are permitted provided that the following conditions are
Lines 103-110 class DeprecatedPort(object): a/Tools/Scripts/webkitpy/common/config/ports.py_sec2
103	command.append("--release")	103	command.append("--release")
104	return command	104	return command
105		105
106	def run_javascriptcore_tests_command(self):	106	def build_jsc_command(self, build_style=None):
107	return self.script_shell_command("run-javascriptcore-tests")	107	command = self.script_shell_command("build-jsc")
		108	if build_style == "debug":
		109	command.append("--debug")
		110	return command
		111
		112	def run_javascriptcore_tests_command(self, build_style=None):
		113	command = self.script_shell_command("run-javascriptcore-tests")
		114	command.append("--no-fail-fast")
		115	command.append("--json-output=jsc_test_results.json")
		116	if (build_style == "debug"):
		117	command.append("--debug")
		118	return command
108		119
109	def run_webkit_tests_command(self, build_style=None):	120	def run_webkit_tests_command(self, build_style=None):
110	command = self.script_shell_command("run-webkit-tests")	121	command = self.script_shell_command("run-webkit-tests")
Lines 150-156 class IOSSimulatorWK2Port(DeprecatedPort): a/Tools/Scripts/webkitpy/common/config/ports.py_sec3
150	class MacPort(DeprecatedPort):	161	class MacPort(DeprecatedPort):
151	port_flag_name = "mac"	162	port_flag_name = "mac"
152		163
153	def run_webkit_tests_command(self, build_style=None):	164	def run_webkit_tests_command(self, build_style=None, test_suite=None):
		165	if (test_suite == "jsc"):
		166	return super(MacPort, self).run_javascriptcore_tests_command(build_style)
		167
154	command = super(MacPort, self).run_webkit_tests_command(build_style)	168	command = super(MacPort, self).run_webkit_tests_command(build_style)
155	command.append("--dump-render-tree")	169	command.append("--dump-render-tree")
156	return command	170	return command

Line 0 a/Tools/Scripts/webkitpy/common/net/jsctestresults.py_sec1
- a/Tools/Scripts/webkitpy/common/net/jsctestresults.py +69 lines
		1	# Copyright (C) 2016 Apple Inc. All rights reserved.
		2	#
		3	# Redistribution and use in source and binary forms, with or without
		4	# modification, are permitted provided that the following conditions
		5	# are met:
		6	# 1. Redistributions of source code must retain the above copyright
		7	# notice, this list of conditions and the following disclaimer.
		8	# 2. Redistributions in binary form must reproduce the above copyright
		9	# notice, this list of conditions and the following disclaimer in the
		10	# documentation and/or other materials provided with the distribution.
		11	#
		12	# THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS'' AND
		13	# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
		14	# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
		15	# DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS BE LIABLE FOR
		16	# ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
		17	# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
		18	# SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
		19	# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
		20	# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
		21	# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
		22
		23	import json
		24	import logging
		25
		26	_log = logging.getLogger(__name__)
		27
		28
		29	class JSCTestResults(object):
		30	@classmethod
		31	def results_from_string(cls, string):
		32	if not string:
		33	return None
		34	parsed_results = json.loads(string)
		35
		36	return cls(parsed_results['allApiTestsPassed'], parsed_results['stressTestFailures'])
		37
		38	def __init__(self, all_api_tests_passed, stress_test_failures):
		39	self._all_api_tests_passed = all_api_tests_passed
		40	self._stress_test_failures = stress_test_failures
		41
		42	self._test_results = stress_test_failures
		43	if self._all_api_tests_passed == False:
		44	self._test_results += "apiTests"
		45
		46	def __eq__(self, other):
		47	return self._all_api_tests_passed == other._all_api_tests_passed and set(self._stress_test_failures) == set(other._stress_test_failures)
		48
		49	def stress_test_failures(self):
		50	return self._stress_test_failures
		51
		52	def all_api_tests_passed(self):
		53	return self._all_api_tests_passed
		54
		55	def all_passed(self):
		56	return self._all_api_tests_passed == True and len(self._stress_test_failures) == 0
		57
		58	def failing_test_results(self):
		59	return self.stress_test_failures()
		60
		61	def test_results(self):
		62	return self._test_results
		63
		64	def failing_tests(self):
65	return self._test_results
66
67	# No defined failure limit for JSC
68	def did_exceed_test_failure_limit(self):
69	return False

Line 0 a/Tools/Scripts/webkitpy/tool/bot/jsctestresultsreader.py_sec1
- a/Tools/Scripts/webkitpy/tool/bot/jsctestresultsreader.py +47 lines
	1	# Copyright (C) 2016 Apple Inc. All rights reserved.
	2	#
	3	# Redistribution and use in source and binary forms, with or without
	4	# modification, are permitted provided that the following conditions
	5	# are met:
	6	# 1. Redistributions of source code must retain the above copyright
	7	# notice, this list of conditions and the following disclaimer.
	8	# 2. Redistributions in binary form must reproduce the above copyright
	9	# notice, this list of conditions and the following disclaimer in the
	10	# documentation and/or other materials provided with the distribution.
	11	#
	12	# THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS'' AND
	13	# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
	14	# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
	15	# DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS BE LIABLE FOR
	16	# ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
	17	# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
	18	# SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
	19	# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
	20	# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
	21	# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
	22
	23	import logging
	24	import json
	25	from webkitpy.common.net.jsctestresults import JSCTestResults
	26
	27	_log = logging.getLogger(__name__)
	28
	29
	30	class JSCTestResultsReader(object):
	31	def __init__(self, host, results_directory):
	32	self._host = host
	33	self._results_directory = results_directory
	34
	35	def _read_file_contents(self, path):
	36	try:
	37	return self._host.filesystem.read_text_file(path)
	38	except IOError, e: # File does not exist or can't be read.
	39	return None
	40
	41	def _create_jsc_test_results(self):
	42	results_path = self._host.filesystem.join(self._results_directory, "jsc_test_results.json")
	43	contents = self._read_file_contents(results_path)
	44	return JSCTestResults.results_from_string(contents)
	45
	46	def results(self):
	47	return self._create_jsc_test_results()

Lines 1-4 a/Tools/Scripts/webkitpy/tool/bot/patchanalysistask.py_sec1
- a/Tools/Scripts/webkitpy/tool/bot/patchanalysistask.py -65 / +109 lines
1	# Copyright (c) 2010 Google Inc. All rights reserved.	1	# Copyright (c) 2010 Google Inc. All rights reserved.
		2	# Copyright (c) 2016 Apple Inc. All rights reserved.
2	#	3	#
3	# Redistribution and use in source and binary forms, with or without	4	# Redistribution and use in source and binary forms, with or without
4	# modification, are permitted provided that the following conditions are	5	# modification, are permitted provided that the following conditions are
Lines 123-149 class PatchAnalysisTask(object): a/Tools/Scripts/webkitpy/tool/bot/patchanalysistask.py_sec2
123	"Patch does not apply")	124	"Patch does not apply")
124		125
125	def _build(self):	126	def _build(self):
126	return self._run_command([	127	args = [
127	"build",	128	"build",
128	"--no-clean",	129	"--no-clean",
129	"--no-update",	130	"--no-update",
130	"--build-style=%s" % self._delegate.build_style(),	131	"--build-style=%s" % self._delegate.build_style(),
131	],	132	]
132	"Built patch",	133
133	"Patch does not build")	134	try:
		135	args.append("--group=%s" % self._delegate.group())
		136	finally:
		137	return self._run_command(
		138	args,
		139	"Built patch",
		140	"Patch does not build")
134		141
135	def _build_without_patch(self):	142	def _build_without_patch(self):
136	return self._run_command([	143	args = [
137	"build",	144	"build",
138	"--force-clean",	145	"--force-clean",
139	"--no-update",	146	"--no-update",
140	"--build-style=%s" % self._delegate.build_style(),	147	"--build-style=%s" % self._delegate.build_style(),
141	],	148	]
142	"Able to build without patch",	149
143	"Unable to build without patch")	150	try:
		151	args.append("--group=%s" % self._delegate.group())
		152	finally:
		153	return self._run_command(
		154	args,
		155	"Able to build without patch",
		156	"Unable to build without patch")
144		157
145	def _test(self):	158	def _test(self):
146	return self._run_command([	159	args = [
147	"build-and-test",	160	"build-and-test",
148	"--no-clean",	161	"--no-clean",
149	"--no-update",	162	"--no-update",
Lines 151-162 class PatchAnalysisTask(object): a/Tools/Scripts/webkitpy/tool/bot/patchanalysistask.py_sec3
151	"--test",	164	"--test",
152	"--non-interactive",	165	"--non-interactive",
153	"--build-style=%s" % self._delegate.build_style(),	166	"--build-style=%s" % self._delegate.build_style(),
154	],	167	]
155	"Passed tests",	168
156	"Patch does not pass tests")	169	try:
		170	args.append("--group=%s" % self._delegate.group())
		171	finally:
		172	return self._run_command(
		173	args,
		174	"Passed tests",
		175	"Patch does not pass tests")
157		176
158	def _build_and_test_without_patch(self):	177	def _build_and_test_without_patch(self):
159	return self._run_command([	178	args = [
160	"build-and-test",	179	"build-and-test",
161	"--force-clean",	180	"--force-clean",
162	"--no-update",	181	"--no-update",
Lines 164-172 class PatchAnalysisTask(object): a/Tools/Scripts/webkitpy/tool/bot/patchanalysistask.py_sec4
164	"--test",	183	"--test",
165	"--non-interactive",	184	"--non-interactive",
166	"--build-style=%s" % self._delegate.build_style(),	185	"--build-style=%s" % self._delegate.build_style(),
167	],	186	]
168	"Able to pass tests without patch",	187
169	"Unable to pass tests without patch (tree is red?)")	188	try:
		189	args.append("--group=%s" % self._delegate.group())
		190	finally:
		191	return self._run_command(
		192	args,
		193	"Able to pass tests without patch",
		194	"Unable to pass tests without patch (tree is red?)")
170		195
171	def _land(self):	196	def _land(self):
172	# Unclear if this should pass --quiet or not. If --parent-command always does the reporting, then it should.	197	# Unclear if this should pass --quiet or not. If --parent-command always does the reporting, then it should.
Lines 210-278 class PatchAnalysisTask(object): a/Tools/Scripts/webkitpy/tool/bot/patchanalysistask.py_sec5
210	if self._test():	235	if self._test():
211	return True	236	return True
212		237
213	# Note: archive_last_test_results deletes the results directory, making these calls order-sensitve.	238	if hasattr(self._delegate, 'group') and self._delegate.group() == "jsc":
214	# We could remove this dependency by building the test_results from the archive.	239	first_results = self._delegate.test_results()
215	first_results = self._delegate.test_results()	240	first_script_error = self._script_error
216	first_results_archive = self._delegate.archive_last_test_results(self._patch)	241	if first_results.all_api_tests_passed() is True and len(first_results.stress_test_failures()) == 0:
217	first_script_error = self._script_error	242	return True
218	first_failure_status_id = self.failure_status_id	243	else:
219		244	# TODO svijayaraghavan@apple.com: Comment this, and uncomment the below lines so it actually builds without patch
220	if self._test() and not first_results.did_exceed_test_failure_limit():	245	#clean_tree_results = first_results
221	# Only report flaky tests if we were successful at parsing results.json and archiving results.	246	#clean_tree_results['allApiTestsPassed'] = False
222	if first_results and first_results_archive:
223	self._report_flaky_tests(first_results.failing_test_results(), first_results_archive)
224	return True
225		247
226	second_results = self._delegate.test_results()	248	self._build_and_test_without_patch()
227	second_results_archive = self._delegate.archive_last_test_results(self._patch)	249	clean_tree_results = self._delegate.test_results()
228	second_script_error = self._script_error
229	second_failure_status_id = self.failure_status_id
230		250
231	if second_results.did_exceed_test_failure_limit() and first_results.did_exceed_test_failure_limit():	251	if clean_tree_results == first_results:
232	self._build_and_test_without_patch()	252	return True
233	clean_tree_results = self._delegate.test_results()
234		253
235	if (len(first_results.failing_tests()) - len(clean_tree_results.failing_tests())) <= 5:	254	return self.report_failure(None, first_results, first_script_error)
236	return False
237		255
238	self.failure_status_id = first_failure_status_id	256	else:
		257	# Note: archive_last_test_results deletes the results directory, making these calls order-sensitve.
		258	# We could remove this dependency by building the test_results from the archive.
		259	first_results = self._delegate.test_results()
		260	first_results_archive = self._delegate.archive_last_test_results(self._patch)
		261	first_script_error = self._script_error
		262	first_failure_status_id = self.failure_status_id
239		263
240	return self.report_failure(first_results_archive, first_results, first_script_error)	264	if self._test() and not first_results.did_exceed_test_failure_limit():
		265	# Only report flaky tests if we were successful at parsing results.json and archiving results.
		266	if first_results and first_results_archive:
		267	self._report_flaky_tests(first_results.failing_test_results(), first_results_archive)
		268	return True
241		269
242	if second_results.did_exceed_test_failure_limit():	270	second_results = self._delegate.test_results()
243	self._should_defer_patch_or_throw(first_results.failing_test_results(), first_results_archive, first_script_error, first_failure_status_id)	271	second_results_archive = self._delegate.archive_last_test_results(self._patch)
244	return False	272	second_script_error = self._script_error
		273	second_failure_status_id = self.failure_status_id
245		274
246	if first_results.did_exceed_test_failure_limit():	275	if second_results.did_exceed_test_failure_limit() and first_results.did_exceed_test_failure_limit():
247	self._should_defer_patch_or_throw(second_results.failing_test_results(), second_results_archive, second_script_error, second_failure_status_id)	276	self._build_and_test_without_patch()
248	return False	277	clean_tree_results = self._delegate.test_results()
		278
		279	if (len(first_results.failing_tests()) - len(clean_tree_results.failing_tests())) <= 5:
		280	return False
249		281
250	if self._results_failed_different_tests(first_results, second_results):	282	self.failure_status_id = first_failure_status_id
251	first_failing_results_set = frozenset(first_results.failing_test_results())
252	second_failing_results_set = frozenset(second_results.failing_test_results())
253		283
254	tests_that_only_failed_first = first_failing_results_set.difference(second_failing_results_set)	284	return self.report_failure(first_results_archive, first_results, first_script_error)
255	self._report_flaky_tests(tests_that_only_failed_first, first_results_archive)
256		285
257	tests_that_only_failed_second = second_failing_results_set.difference(first_failing_results_set)	286	if second_results.did_exceed_test_failure_limit():
258	self._report_flaky_tests(tests_that_only_failed_second, second_results_archive)	287	self._should_defer_patch_or_throw(first_results.failing_test_results(), first_results_archive, first_script_error, first_failure_status_id)
		288	return False
		289
		290	if first_results.did_exceed_test_failure_limit():
		291	self._should_defer_patch_or_throw(second_results.failing_test_results(), second_results_archive, second_script_error, second_failure_status_id)
		292	return False
		293
		294	if self._results_failed_different_tests(first_results, second_results):
		295	first_failing_results_set = frozenset(first_results.failing_test_results())
		296	second_failing_results_set = frozenset(second_results.failing_test_results())
259		297
260	tests_that_consistently_failed = first_failing_results_set.intersection(second_failing_results_set)	298	tests_that_only_failed_first = first_failing_results_set.difference(second_failing_results_set)
261	if tests_that_consistently_failed:	299	self._report_flaky_tests(tests_that_only_failed_first, first_results_archive)
262	if self._should_defer_patch_or_throw(tests_that_consistently_failed, first_results_archive, first_script_error, first_failure_status_id):
263	return False # Defer patch
264		300
265	# At this point we know that at least one test flaked, but no consistent failures	301	tests_that_only_failed_second = second_failing_results_set.difference(first_failing_results_set)
266	# were introduced. This is a bit of a grey-zone.	302	self._report_flaky_tests(tests_that_only_failed_second, second_results_archive)
267	return False # Defer patch
268		303
269	if self._should_defer_patch_or_throw(first_results.failing_test_results(), first_results_archive, first_script_error, first_failure_status_id):	304	tests_that_consistently_failed = first_failing_results_set.intersection(second_failing_results_set)
270	return False # Defer patch	305	if tests_that_consistently_failed:
		306	if self._should_defer_patch_or_throw(tests_that_consistently_failed, first_results_archive, first_script_error, first_failure_status_id):
		307	return False # Defer patch
271		308
272	# At this point, we know that the first and second runs had the exact same failures,	309	# At this point we know that at least one test flaked, but no consistent failures
273	# and that those failures are all present on the clean tree, so we can say with certainty	310	# were introduced. This is a bit of a grey-zone.
274	# that the patch is good.	311	return False # Defer patch
275	return True	312
		313	if self._should_defer_patch_or_throw(first_results.failing_test_results(), first_results_archive, first_script_error, first_failure_status_id):
		314	return False # Defer patch
		315
		316	# At this point, we know that the first and second runs had the exact same failures,
		317	# and that those failures are all present on the clean tree, so we can say with certainty
		318	# that the patch is good.
		319	return True
276		320
277	def results_archive_from_patch_test_run(self, patch):	321	def results_archive_from_patch_test_run(self, patch):
278	assert(self._patch.id() == patch.id()) # PatchAnalysisTask is not currently re-useable.	322	assert(self._patch.id() == patch.id()) # PatchAnalysisTask is not currently re-useable.

Lines 1-4 a/Tools/Scripts/webkitpy/tool/steps/build.py_sec1
- a/Tools/Scripts/webkitpy/tool/steps/build.py -7 / +18 lines
1	# Copyright (C) 2010 Google Inc. All rights reserved.	1	# Copyright (C) 2010 Google Inc. All rights reserved.
		2	# Copyright (C) 2016 Apple Inc. All rights reserved.
2	#	3	#
3	# Redistribution and use in source and binary forms, with or without	4	# Redistribution and use in source and binary forms, with or without
4	# modification, are permitted provided that the following conditions are	5	# modification, are permitted provided that the following conditions are
Lines 41-68 class Build(AbstractStep): a/Tools/Scripts/webkitpy/tool/steps/build.py_sec2
41	Options.build,	42	Options.build,
42	Options.quiet,	43	Options.quiet,
43	Options.build_style,	44	Options.build_style,
		45	Options.group,
44	]	46	]
45		47
46	def build(self, build_style):	48	def build(self, build_style, group):
47	environment = self._tool.copy_current_environment()	49	environment = self._tool.copy_current_environment()
48	environment.disable_gcc_smartquotes()	50	environment.disable_gcc_smartquotes()
49	environment.disable_jhbuild_VT100_output()	51	environment.disable_jhbuild_VT100_output()
50	env = environment.to_dictionary()	52	env = environment.to_dictionary()
51		53
52	build_webkit_command = self._tool.deprecated_port().build_webkit_command(build_style=build_style)	54	if group == "jsc":
		55	build_command = self._tool.deprecated_port().build_jsc_command(build_style=build_style)
		56	else:
		57	build_command = self._tool.deprecated_port().build_webkit_command(build_style=build_style)
53		58
54	if self._options.architecture:	59	if self._options.architecture:
55	build_webkit_command += ['ARCHS=%s' % self._options.architecture]	60	build_command += ['ARCHS=%s' % self._options.architecture]
56		61
57	self._tool.executive.run_and_throw_if_fail(build_webkit_command, self._options.quiet,	62	self._tool.executive.run_and_throw_if_fail(build_command, self._options.quiet,
58	cwd=self._tool.scm().checkout_root, env=env)	63	cwd=self._tool.scm().checkout_root, env=env)
59		64
60	def run(self, state):	65	def run(self, state):
61	if not self._options.build:	66	if not self._options.build:
62	return	67	return
63	_log.info("Building WebKit")	68	_log.info("Building WebKit")
		69
		70	if hasattr(self._options, 'group') and self._options.group == "jsc":
		71	group = self._options.group
		72	else:
		73	group = None
		74
64	if self._options.build_style == "both":	75	if self._options.build_style == "both":
65	self.build("debug")	76	self.build("debug", group)
66	self.build("release")	77	self.build("release", group)
67	else:	78	else:
68	self.build(self._options.build_style)	79	self.build(self._options.build_style, group)

Lines 1-4 a/Tools/Scripts/webkitpy/tool/steps/options.py_sec1
- a/Tools/Scripts/webkitpy/tool/steps/options.py +2 lines
1	# Copyright (C) 2010 Google Inc. All rights reserved.	1	# Copyright (C) 2010 Google Inc. All rights reserved.
		2	# Copyright (C) 2016 Apple Inc. All rights reserved.
2	#	3	#
3	# Redistribution and use in source and binary forms, with or without	4	# Redistribution and use in source and binary forms, with or without
4	# modification, are permitted provided that the following conditions are	5	# modification, are permitted provided that the following conditions are
Lines 46-51 class Options(object): a/Tools/Scripts/webkitpy/tool/steps/options.py_sec2
46	email = make_option("--email", action="store", type="string", dest="email", help="Email address to use in ChangeLogs.")	47	email = make_option("--email", action="store", type="string", dest="email", help="Email address to use in ChangeLogs.")
47	force_clean = make_option("--force-clean", action="store_true", dest="force_clean", default=False, help="Clean working directory before applying patches (removes local changes and commits)")	48	force_clean = make_option("--force-clean", action="store_true", dest="force_clean", default=False, help="Clean working directory before applying patches (removes local changes and commits)")
48	git_commit = make_option("-g", "--git-commit", action="store", dest="git_commit", help="Operate on a local commit. If a range, the commits are squashed into one. <ref>.... includes the working copy changes. UPSTREAM can be used for the upstream/tracking branch.")	49	git_commit = make_option("-g", "--git-commit", action="store", dest="git_commit", help="Operate on a local commit. If a range, the commits are squashed into one. <ref>.... includes the working copy changes. UPSTREAM can be used for the upstream/tracking branch.")
		50	group = make_option("--group", action="store", dest="group", default=None, help="")
49	local_commit = make_option("--local-commit", action="store_true", dest="local_commit", default=False, help="Make a local commit for each applied patch")	51	local_commit = make_option("--local-commit", action="store_true", dest="local_commit", default=False, help="Make a local commit for each applied patch")
50	non_interactive = make_option("--non-interactive", action="store_true", dest="non_interactive", default=False, help="Never prompt the user, fail as fast as possible.")	52	non_interactive = make_option("--non-interactive", action="store_true", dest="non_interactive", default=False, help="Never prompt the user, fail as fast as possible.")
51	obsolete_patches = make_option("--no-obsolete", action="store_false", dest="obsolete_patches", default=True, help="Do not obsolete old patches before posting this one.")	53	obsolete_patches = make_option("--no-obsolete", action="store_false", dest="obsolete_patches", default=True, help="Do not obsolete old patches before posting this one.")

Lines 1-4 a/Tools/Scripts/webkitpy/tool/steps/steps_unittest.py_sec1
- a/Tools/Scripts/webkitpy/tool/steps/steps_unittest.py -2 / +5 lines
1	# Copyright (C) 2010 Google Inc. All rights reserved.	1	# Copyright (C) 2010 Google Inc. All rights reserved.
		2	# Copyright (C) 2016 Apple Inc. All rights reserved.
2	#	3	#
3	# Redistribution and use in source and binary forms, with or without	4	# Redistribution and use in source and binary forms, with or without
4	# modification, are permitted provided that the following conditions are	5	# modification, are permitted provided that the following conditions are
Lines 98-103 class StepsTest(unittest.TestCase): a/Tools/Scripts/webkitpy/tool/steps/steps_unittest.py_sec2
98	self._assert_step_output_with_bug(steps.EnsureBugIsOpenAndAssigned, 50002, expected_logs)	99	self._assert_step_output_with_bug(steps.EnsureBugIsOpenAndAssigned, 50002, expected_logs)
99		100
100	def test_runtests_args(self):	101	def test_runtests_args(self):
		102	self.maxDiff = None
101	mock_options = self._step_options()	103	mock_options = self._step_options()
102	mock_options.non_interactive = False	104	mock_options.non_interactive = False
103	mock_options.build_style = "release"	105	mock_options.build_style = "release"
Lines 111-117 MOCK run_and_throw_if_fail: ['Tools/Scripts/test-webkitpy'], cwd=/mock-checkout a/Tools/Scripts/webkitpy/tool/steps/steps_unittest.py_sec3
111	Running Perl unit tests	113	Running Perl unit tests
112	MOCK run_and_throw_if_fail: ['Tools/Scripts/test-webkitperl'], cwd=/mock-checkout	114	MOCK run_and_throw_if_fail: ['Tools/Scripts/test-webkitperl'], cwd=/mock-checkout
113	Running JavaScriptCore tests	115	Running JavaScriptCore tests
114	MOCK run_and_throw_if_fail: ['Tools/Scripts/run-javascriptcore-tests'], cwd=/mock-checkout	116	MOCK run_and_throw_if_fail: ['Tools/Scripts/run-javascriptcore-tests', '--no-fail-fast', '--json-output=jsc_test_results.json'], cwd=/mock-checkout
115	Running bindings generation tests	117	Running bindings generation tests
116	MOCK run_and_throw_if_fail: ['Tools/Scripts/run-bindings-tests'], cwd=/mock-checkout	118	MOCK run_and_throw_if_fail: ['Tools/Scripts/run-bindings-tests'], cwd=/mock-checkout
117	Running run-webkit-tests	119	Running run-webkit-tests
Lines 120-125 MOCK run_and_throw_if_fail: ['Tools/Scripts/run-webkit-tests', '--release', '--q a/Tools/Scripts/webkitpy/tool/steps/steps_unittest.py_sec4
120	OutputCapture().assert_outputs(self, step.run, [{}], expected_logs=expected_logs)	122	OutputCapture().assert_outputs(self, step.run, [{}], expected_logs=expected_logs)
121		123
122	def test_runtests_debug_args(self):	124	def test_runtests_debug_args(self):
		125	self.maxDiff = None
123	mock_options = self._step_options()	126	mock_options = self._step_options()
124	mock_options.non_interactive = False	127	mock_options.non_interactive = False
125	mock_options.build_style = "debug"	128	mock_options.build_style = "debug"
Lines 133-139 MOCK run_and_throw_if_fail: ['Tools/Scripts/test-webkitpy'], cwd=/mock-checkout a/Tools/Scripts/webkitpy/tool/steps/steps_unittest.py_sec5
133	Running Perl unit tests	136	Running Perl unit tests
134	MOCK run_and_throw_if_fail: ['Tools/Scripts/test-webkitperl'], cwd=/mock-checkout	137	MOCK run_and_throw_if_fail: ['Tools/Scripts/test-webkitperl'], cwd=/mock-checkout
135	Running JavaScriptCore tests	138	Running JavaScriptCore tests
136	MOCK run_and_throw_if_fail: ['Tools/Scripts/run-javascriptcore-tests'], cwd=/mock-checkout	139	MOCK run_and_throw_if_fail: ['Tools/Scripts/run-javascriptcore-tests', '--no-fail-fast', '--json-output=jsc_test_results.json'], cwd=/mock-checkout
137	Running bindings generation tests	140	Running bindings generation tests
138	MOCK run_and_throw_if_fail: ['Tools/Scripts/run-bindings-tests'], cwd=/mock-checkout	141	MOCK run_and_throw_if_fail: ['Tools/Scripts/run-bindings-tests'], cwd=/mock-checkout
139	Running run-webkit-tests	142	Running run-webkit-tests