Attachment #297239 for bug #162458

View Details Raw Unified Return to Bug 162458

| Differences between



2016-12-15  Srinivasan Vijayaraghavan  <svijayaraghavan@apple.com>

        EWS should run JavaScriptCore tests
        https://bugs.webkit.org/show_bug.cgi?id=162458

        Reviewed by NOBODY (OOPS!).

        * Scripts/webkitpy/common/config/ews.json: Add config info for JSC EWS.
        * Scripts/webkitpy/common/config/ports.py: 
        (DeprecatedPort.build_jsc_command): Added - command to build JSC only (quicker than building all of WebKit).
        (DeprecatedPort):
        (DeprecatedPort.run_javascriptcore_tests_command): Allow JSC EWS to only run JSC tests.
        (MacPort.run_webkit_tests_command):
        * Scripts/webkitpy/tool/bot/patchanalysistask.py:
        (PatchAnalysisTask._build): Check for JSC.
        (PatchAnalysisTask._build_without_patch): Check for JSC.
        (PatchAnalysisTask._test): Check for JSC.
        (PatchAnalysisTask._build_and_test_without_patch): Check for JSC.
        (PatchAnalysisTask._test_patch): Add retry logic for JSC.
        * Scripts/webkitpy/tool/commands/earlywarningsystem.py:
        (AbstractEarlyWarningSystem.begin_work_queue): Use JSCTestResultsReader instead of LayoutTestResultsReader in JSC EWS.
        (AbstractEarlyWarningSystem.test_results): _layout_test_results_reader -> _test_results_reader.
        (AbstractEarlyWarningSystem.archive_last_test_results): _layout_test_results_reader -> _test_results_reader.
        (AbstractEarlyWarningSystem.group): This attribute determines the type of EWS (eg. JSC).
        (AbstractEarlyWarningSystem.load_ews_classes): Add _group.
        * Scripts/webkitpy/tool/commands/earlywarningsystem_unittest.py:
        (EarlyWarningSystemTest._default_expected_logs): Add _group.
        * Scripts/webkitpy/tool/steps/build.py:
        (Build.options): Check for JSC.
        (Build.build): Check for JSC.
        (Build.run): Check for JSC.
        * Scripts/webkitpy/tool/steps/options.py:
        (Options): Add --group command line option.
        * Scripts/webkitpy/tool/steps/runtests.py:
        (RunTests.options): 
        (RunTests.run): Check for JSC.
        (RunTests._run_webkit_tests): Check for JSC.
        (RunTests): Add _group attribute.
        (RunTests._run_javascriptcore_tests): New.
        * Scripts/webkitpy/tool/steps/steps_unittest.py: Change expected command for running jsc tests.

2016-09-25  Gyuyoung Kim  <gyuyoung.kim@webkit.org>

        [EFL] Adjust EFL indentation to EFL MiniBrowser

Lines 50-54 a/Tools/Scripts/webkitpy/common/config/ews.json_sec1
- a/Tools/Scripts/webkitpy/common/config/ews.json +6 lines
50	"port": "mac",	50	"port": "mac",
51	"name": "mac-32bit-ews",	51	"name": "mac-32bit-ews",
52	"architecture": "i386"	52	"architecture": "i386"
		53	},
		54	"JSC EWS": {
		55	"port": "mac",
		56	"name": "jsc-ews",
		57	"group": "jsc",
		58	"runTests": true
53	}	59	}
54	}	60	}



            command.append("--release")
        return command

    def build_jsc_command(self, build_style=None):
        command = self.script_shell_command("build-jsc")
        if build_style == "debug":
            command.append("--debug")
        return command

    def run_javascriptcore_tests_command(self, build_style=None):
        command = self.script_shell_command("run-javascriptcore-tests")
        command.append("--no-fail-fast")
        command.append("--json-output=jsc_test_results.json")
        command.append("--no-mozilla-tests")
        command.append("--no-build")
        if (build_style == "debug"):
            command.append("--debug")
        return command

    def run_webkit_tests_command(self, build_style=None):
        command = self.script_shell_command("run-webkit-tests")

class MacPort(DeprecatedPort):
    port_flag_name = "mac"

    def run_webkit_tests_command(self, build_style=None, test_suite=None):
        if (test_suite == "jsc"):
            return super(MacPort, self).run_javascriptcore_tests_command(build_style)

        command = super(MacPort, self).run_webkit_tests_command(build_style)
        command.append("--dump-render-tree")
        return command



# Copyright (C) 2016 Apple Inc. All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions
# are met:
# 1.  Redistributions of source code must retain the above copyright
#     notice, this list of conditions and the following disclaimer.
# 2.  Redistributions in binary form must reproduce the above copyright
#     notice, this list of conditions and the following disclaimer in the
#     documentation and/or other materials provided with the distribution.
#
# THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS'' AND
# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
# DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS BE LIABLE FOR
# ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
# SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

import json
import logging
from webkitpy.thirdparty.BeautifulSoup import BeautifulSoup, SoupStrainer

_log = logging.getLogger(__name__)


class JSCTestResults(object):
    @classmethod
    def results_from_string(cls, string):
        if not string:
            return None
        parsed_results = json.load(string)
        return cls(parsed_results['allApiTestsPassed'], parsed_results['stressTestFailures'])

    def __init__(self, all_api_tests_passed, stress_test_failures):
        self._all_api_tests_passed = all_api_tests_passed
        self._stress_test_failures = stress_test_failures

        self._test_results = stress_test_failures
        if self._all_api_tests_passed == False:
            self._test_results += "apiTests"

    def stress_test_failures(self):
        return self._stress_test_failures

    def all_api_tests_passed(self):
        return self._all_api_tests_passed

    def all_passed(self):
        return self._all_api_tests_passed == True and len(self._stress_test_failures) == 0

    def failing_test_results(self):
        return self.stress_test_failures()

    def test_results(self):
        return self._test_results

    def failing_tests(self):
        return self._test_results

    # No defined failure limit for JSC
    def did_exceed_test_failure_limit(self):
        return False



# Copyright (C) 2016 Apple Inc. All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions
# are met:
# 1.  Redistributions of source code must retain the above copyright
#     notice, this list of conditions and the following disclaimer.
# 2.  Redistributions in binary form must reproduce the above copyright
#     notice, this list of conditions and the following disclaimer in the
#     documentation and/or other materials provided with the distribution.
#
# THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS'' AND
# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
# DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS BE LIABLE FOR
# ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
# SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

import logging
import json
from webkitpy.common.net.jsctestresults import JSCTestResults

_log = logging.getLogger(__name__)


class JSCTestResultsReader(object):
    def __init__(self, host, results_directory):
        self._host = host
        self._results_directory = results_directory

    def _read_file_contents(self, path):
        try:
            return self._host.filesystem.read_text_file(path)
        except IOError, e:  # File does not exist or can't be read.
            return None

    def _create_jsc_test_results(self):
        results_path = self._host.filesystem.join(self._results_directory, "jsc_test_results.json")
        contents = open(results_path)
        return JSCTestResults.results_from_string(contents)

    def results(self):
        return self._create_jsc_test_results()



        "Patch does not apply")

    def _build(self):
        args = [
            "build",
            "--no-clean",
            "--no-update",
            "--build-style=%s" % self._delegate.build_style(),
        ]

        try:
            args.append("--group=%s" % self._delegate.group())
        finally:
            return self._run_command(
                args,
                "Built patch",
                "Patch does not build")

    def _build_without_patch(self):
        args = [
            "build",
            "--force-clean",
            "--no-update",
            "--build-style=%s" % self._delegate.build_style(),
        ]

        try:
            args.append("--group=%s" % self._delegate.group())
        finally:
            return self._run_command(
                args,
                "Able to build without patch",
                "Unable to build without patch")

    def _test(self):
        args = [
            "build-and-test",
            "--no-clean",
            "--no-update",

            "--test",
            "--non-interactive",
            "--build-style=%s" % self._delegate.build_style(),
        ]

        try:
            args.append("--group=%s" % self._delegate.group())
        finally:
            return self._run_command(
                args,
                "Passed tests",
                "Patch does not pass tests")

    def _build_and_test_without_patch(self):
        args = [
            "build-and-test",
            "--force-clean",
            "--no-update",

            "--test",
            "--non-interactive",
            "--build-style=%s" % self._delegate.build_style(),
        ]

        try:
            args.append("--group=%s" % self._delegate.group())
        finally:
            return self._run_command(
                args,
                "Able to pass tests without patch",
                "Unable to pass tests without patch (tree is red?)")

    def _land(self):
        # Unclear if this should pass --quiet or not.  If --parent-command always does the reporting, then it should.

        if self._test():
            return True

        if hasattr(self._delegate, 'group') and self._delegate.group() == "jsc":
            first_results = self._delegate.test_results()
            first_script_error = self._script_error
            if first_results.all_api_tests_passed() is True and len(first_results.stress_test_failures()) == 0:
                return True
            else:
                # TODO svijayaraghavan@apple.com: Comment this, and uncomment the below lines so it actually builds without patch
                #clean_tree_results = first_results
                #clean_tree_results['allApiTestsPassed'] = False
            if first_results and first_results_archive:
                self._report_flaky_tests(first_results.failing_test_results(), first_results_archive)
            return True

                self._build_and_test_without_patch()
                clean_tree_results = self._delegate.test_results()
        second_script_error = self._script_error
        second_failure_status_id = self.failure_status_id

                if clean_tree_results == first_results:
                    return True
            clean_tree_results = self._delegate.test_results()

                return self.report_failure(None, first_results, first_script_error)
                return False

        else:
            # Note: archive_last_test_results deletes the results directory, making these calls order-sensitve.
            # We could remove this dependency by building the test_results from the archive.
            first_results = self._delegate.test_results()
            first_results_archive = self._delegate.archive_last_test_results(self._patch)
            first_script_error = self._script_error
            first_failure_status_id = self.failure_status_id

            if self._test() and not first_results.did_exceed_test_failure_limit():
                # Only report flaky tests if we were successful at parsing results.json and archiving results.
                if first_results and first_results_archive:
                    self._report_flaky_tests(first_results.failing_test_results(), first_results_archive)
                return True

            second_results = self._delegate.test_results()
            second_results_archive = self._delegate.archive_last_test_results(self._patch)
            second_script_error = self._script_error
            second_failure_status_id = self.failure_status_id

            if second_results.did_exceed_test_failure_limit() and first_results.did_exceed_test_failure_limit():
                self._build_and_test_without_patch()
                clean_tree_results = self._delegate.test_results()

                if (len(first_results.failing_tests()) - len(clean_tree_results.failing_tests())) <= 5:
                    return False

                self.failure_status_id = first_failure_status_id
            first_failing_results_set = frozenset(first_results.failing_test_results())
            second_failing_results_set = frozenset(second_results.failing_test_results())

                return self.report_failure(first_results_archive, first_results, first_script_error)
            self._report_flaky_tests(tests_that_only_failed_first, first_results_archive)

            if second_results.did_exceed_test_failure_limit():
                self._should_defer_patch_or_throw(first_results.failing_test_results(), first_results_archive, first_script_error, first_failure_status_id)
                return False

            if first_results.did_exceed_test_failure_limit():
                self._should_defer_patch_or_throw(second_results.failing_test_results(), second_results_archive, second_script_error, second_failure_status_id)
                return False

            if self._results_failed_different_tests(first_results, second_results):
                first_failing_results_set = frozenset(first_results.failing_test_results())
                second_failing_results_set = frozenset(second_results.failing_test_results())

                tests_that_only_failed_first = first_failing_results_set.difference(second_failing_results_set)
                self._report_flaky_tests(tests_that_only_failed_first, first_results_archive)
                if self._should_defer_patch_or_throw(tests_that_consistently_failed, first_results_archive, first_script_error, first_failure_status_id):
                    return False  # Defer patch

                tests_that_only_failed_second = second_failing_results_set.difference(first_failing_results_set)
                self._report_flaky_tests(tests_that_only_failed_second, second_results_archive)
            return False  # Defer patch

                tests_that_consistently_failed = first_failing_results_set.intersection(second_failing_results_set)
                if tests_that_consistently_failed:
                    if self._should_defer_patch_or_throw(tests_that_consistently_failed, first_results_archive, first_script_error, first_failure_status_id):
                        return False  # Defer patch

                # At this point we know that at least one test flaked, but no consistent failures
                # were introduced. This is a bit of a grey-zone.
                return False  # Defer patch

            if self._should_defer_patch_or_throw(first_results.failing_test_results(), first_results_archive, first_script_error, first_failure_status_id):
                return False  # Defer patch

            # At this point, we know that the first and second runs had the exact same failures,
            # and that those failures are all present on the clean tree, so we can say with certainty
            # that the patch is good.
            return True

    def results_archive_from_patch_test_run(self, patch):
        assert(self._patch.id() == patch.id())  # PatchAnalysisTask is not currently re-useable.

Lines 28-33 a/Tools/Scripts/webkitpy/tool/commands/earlywarningsystem.py_sec1
- a/Tools/Scripts/webkitpy/tool/commands/earlywarningsystem.py -3 / +17 lines
28		28
29	import json	29	import json
30	import logging	30	import logging
		31	import os
31	from optparse import make_option	32	from optparse import make_option
32		33
33	from webkitpy.common.config.committers import CommitterList	34	from webkitpy.common.config.committers import CommitterList
Lines 36-41 from webkitpy.common.system.filesystem import FileSystem a/Tools/Scripts/webkitpy/tool/commands/earlywarningsystem.py_sec2
36	from webkitpy.common.system.executive import ScriptError	37	from webkitpy.common.system.executive import ScriptError
37	from webkitpy.tool.bot.earlywarningsystemtask import EarlyWarningSystemTask, EarlyWarningSystemTaskDelegate	38	from webkitpy.tool.bot.earlywarningsystemtask import EarlyWarningSystemTask, EarlyWarningSystemTaskDelegate
38	from webkitpy.tool.bot.layouttestresultsreader import LayoutTestResultsReader	39	from webkitpy.tool.bot.layouttestresultsreader import LayoutTestResultsReader
		40	from webkitpy.tool.bot.jsctestresultsreader import JSCTestResultsReader
39	from webkitpy.tool.bot.patchanalysistask import UnableToApplyPatch, PatchIsNotValid	41	from webkitpy.tool.bot.patchanalysistask import UnableToApplyPatch, PatchIsNotValid
40	from webkitpy.tool.bot.queueengine import QueueEngine	42	from webkitpy.tool.bot.queueengine import QueueEngine
41	from webkitpy.tool.commands.queues import AbstractReviewQueue	43	from webkitpy.tool.commands.queues import AbstractReviewQueue
Lines 53-59 class AbstractEarlyWarningSystem(AbstractReviewQueue, EarlyWarningSystemTaskDele a/Tools/Scripts/webkitpy/tool/commands/earlywarningsystem.py_sec3
53		55
54	def begin_work_queue(self):	56	def begin_work_queue(self):
55	AbstractReviewQueue.begin_work_queue(self)	57	AbstractReviewQueue.begin_work_queue(self)
56	self._layout_test_results_reader = LayoutTestResultsReader(self._tool, self._port.results_directory(), self._log_directory())	58
		59	if self.group() == "jsc":
		60	dir = os.path.dirname(os.path.dirname(os.path.dirname(self._port.results_directory())))
		61	self._test_results_reader = JSCTestResultsReader(self._tool, dir)
		62	else:
		63	self._test_results_reader = LayoutTestResultsReader(self._tool, self._port.results_directory(), self._log_directory())
57		64
58	def _failing_tests_message(self, task, patch):	65	def _failing_tests_message(self, task, patch):
59	results = task.results_from_patch_test_run(patch)	66	results = task.results_from_patch_test_run(patch)
Lines 117-130 class AbstractEarlyWarningSystem(AbstractReviewQueue, EarlyWarningSystemTaskDele a/Tools/Scripts/webkitpy/tool/commands/earlywarningsystem.py_sec4
117	return self._update_status(message, patch=patch, results_file=failure_log)	124	return self._update_status(message, patch=patch, results_file=failure_log)
118		125
119	def test_results(self):	126	def test_results(self):
120	return self._layout_test_results_reader.results()	127	return self._test_results_reader.results()
121		128
122	def archive_last_test_results(self, patch):	129	def archive_last_test_results(self, patch):
123	return self._layout_test_results_reader.archive(patch)	130	return self._test_results_reader.archive(patch)
124		131
125	def build_style(self):	132	def build_style(self):
126	return self._build_style	133	return self._build_style
127		134
		135	def group(self):
		136	if hasattr(self, "_group"):
		137	return self._group
		138	else:
		139	return None
		140
128	def refetch_patch(self, patch):	141	def refetch_patch(self, patch):
129	return self._tool.bugs.fetch_attachment(patch.id())	142	return self._tool.bugs.fetch_attachment(patch.id())
130		143
Lines 156-160 class AbstractEarlyWarningSystem(AbstractReviewQueue, EarlyWarningSystemTaskDele a/Tools/Scripts/webkitpy/tool/commands/earlywarningsystem.py_sec5
156	'_build_style': config.get('style', "release"),	169	'_build_style': config.get('style', "release"),
157	'watchers': config.get('watchers', []),	170	'watchers': config.get('watchers', []),
158	'run_tests': config.get('runTests', cls.run_tests),	171	'run_tests': config.get('runTests', cls.run_tests),
		172	'_group': config.get('group', None),
159	}))	173	}))
160	return classes	174	return classes



            "port": ews.port_name,
            "architecture": " --architecture=%s" % ews.architecture if ews.architecture else "",
            "build_style": ews.build_style(),
            "group": "jsc" if ews.name == "jsc-ews" else "None",
        }
        if ews.run_tests:
            run_tests_line = "Running: webkit-patch --status-host=example.com build-and-test --no-clean --no-update --test --non-interactive --build-style=%(build_style)s --group=%(group)s --port=%(port)s%(architecture)s\n" % string_replacements
        else:
            run_tests_line = ""
        string_replacements['run_tests_line'] = run_tests_line

Running: webkit-patch --status-host=example.com clean --port=%(port)s%(architecture)s
Running: webkit-patch --status-host=example.com update --port=%(port)s%(architecture)s
Running: webkit-patch --status-host=example.com apply-attachment --no-update --non-interactive 10000 --port=%(port)s%(architecture)s
Running: webkit-patch --status-host=example.com build --no-clean --no-update --build-style=%(build_style)s --group=%(group)s --port=%(port)s%(architecture)s
%(run_tests_line)sMOCK: update_status: %(name)s Pass
MOCK: release_work_item: %(name)s 10000
""" % string_replacements,



            Options.build,
            Options.quiet,
            Options.build_style,
            Options.group,
        ]

    def build(self, build_style, group):
        environment = self._tool.copy_current_environment()
        environment.disable_gcc_smartquotes()
        environment.disable_jhbuild_VT100_output()
        env = environment.to_dictionary()

        if group == "jsc":
            build_command = self._tool.deprecated_port().build_jsc_command(build_style=build_style)
        else:
            build_command = self._tool.deprecated_port().build_webkit_command(build_style=build_style)

        if self._options.architecture:
            build_command += ['ARCHS=%s' % self._options.architecture]

        self._tool.executive.run_and_throw_if_fail(build_command, self._options.quiet,
            cwd=self._tool.scm().checkout_root, env=env)

    def run(self, state):
        if not self._options.build:
            return
        _log.info("Building WebKit")

        if hasattr(self._options, 'group') and self._options.group == "jsc":
            group = self._options.group
        else:
            group = None

        if self._options.build_style == "both":
            self.build("debug", group)
            self.build("release", group)
        else:
            self.build(self._options.build_style, group)

Lines 46-51 class Options(object): a/Tools/Scripts/webkitpy/tool/steps/options.py_sec1
- a/Tools/Scripts/webkitpy/tool/steps/options.py +1 lines
46	email = make_option("--email", action="store", type="string", dest="email", help="Email address to use in ChangeLogs.")	46	email = make_option("--email", action="store", type="string", dest="email", help="Email address to use in ChangeLogs.")
47	force_clean = make_option("--force-clean", action="store_true", dest="force_clean", default=False, help="Clean working directory before applying patches (removes local changes and commits)")	47	force_clean = make_option("--force-clean", action="store_true", dest="force_clean", default=False, help="Clean working directory before applying patches (removes local changes and commits)")
48	git_commit = make_option("-g", "--git-commit", action="store", dest="git_commit", help="Operate on a local commit. If a range, the commits are squashed into one. <ref>.... includes the working copy changes. UPSTREAM can be used for the upstream/tracking branch.")	48	git_commit = make_option("-g", "--git-commit", action="store", dest="git_commit", help="Operate on a local commit. If a range, the commits are squashed into one. <ref>.... includes the working copy changes. UPSTREAM can be used for the upstream/tracking branch.")
		49	group = make_option("--group", action="store", dest="group", default=None, help="")
49	local_commit = make_option("--local-commit", action="store_true", dest="local_commit", default=False, help="Make a local commit for each applied patch")	50	local_commit = make_option("--local-commit", action="store_true", dest="local_commit", default=False, help="Make a local commit for each applied patch")
50	non_interactive = make_option("--non-interactive", action="store_true", dest="non_interactive", default=False, help="Never prompt the user, fail as fast as possible.")	51	non_interactive = make_option("--non-interactive", action="store_true", dest="non_interactive", default=False, help="Never prompt the user, fail as fast as possible.")
51	obsolete_patches = make_option("--no-obsolete", action="store_false", dest="obsolete_patches", default=True, help="Do not obsolete old patches before posting this one.")	52	obsolete_patches = make_option("--no-obsolete", action="store_false", dest="obsolete_patches", default=True, help="Do not obsolete old patches before posting this one.")

Lines 1-4 a/Tools/Scripts/webkitpy/tool/steps/runtests.py_sec1
- a/Tools/Scripts/webkitpy/tool/steps/runtests.py +10 lines
1	# Copyright (C) 2010 Google Inc. All rights reserved.	1	# Copyright (C) 2010 Google Inc. All rights reserved.
		2	# Copyright (C) 2016 Apple Inc. All rights reserved.
2	#	3	#
3	# Redistribution and use in source and binary forms, with or without	4	# Redistribution and use in source and binary forms, with or without
4	# modification, are permitted provided that the following conditions are	5	# modification, are permitted provided that the following conditions are
Lines 53-58 class RunTests(AbstractStep): a/Tools/Scripts/webkitpy/tool/steps/runtests.py_sec2
53	Options.iterate_on_new_tests,	54	Options.iterate_on_new_tests,
54	Options.non_interactive,	55	Options.non_interactive,
55	Options.quiet,	56	Options.quiet,
		57	Options.group,
56	]	58	]
57		59
58	def run(self, state):	60	def run(self, state):
Lines 63-68 class RunTests(AbstractStep): a/Tools/Scripts/webkitpy/tool/steps/runtests.py_sec3
63	if not self._options.test:	65	if not self._options.test:
64	return	66	return
65		67
		68	if hasattr(self._options, "group") and self._options.group == "jsc":
		69	self._run_javascriptcore_tests()
		70	return
		71
66	python_unittests_command = self._tool.deprecated_port().run_python_unittests_command()	72	python_unittests_command = self._tool.deprecated_port().run_python_unittests_command()
67	if python_unittests_command:	73	if python_unittests_command:
68	_log.info("Running Python unit tests")	74	_log.info("Running Python unit tests")
Lines 143-145 class RunTests(AbstractStep): a/Tools/Scripts/webkitpy/tool/steps/runtests.py_sec4
143	args.append("--iterations=%d" % iterate_on_new_tests)	149	args.append("--iterations=%d" % iterate_on_new_tests)
144		150
145	self._tool.executive.run_and_throw_if_fail(args, cwd=self._tool.scm().checkout_root)	151	self._tool.executive.run_and_throw_if_fail(args, cwd=self._tool.scm().checkout_root)
		152
		153	def _run_javascriptcore_tests(self):
		154	args = self._tool.deprecated_port().run_javascriptcore_tests_command(self._options.build_style)
		155	self._tool.executive.run_and_throw_if_fail(args, cwd=self._tool.scm().checkout_root)



        self._assert_step_output_with_bug(steps.EnsureBugIsOpenAndAssigned, 50002, expected_logs)

    def test_runtests_args(self):
        self.maxDiff = None
        mock_options = self._step_options()
        mock_options.non_interactive = False
        mock_options.build_style = "release"

Running Perl unit tests
MOCK run_and_throw_if_fail: ['Tools/Scripts/test-webkitperl'], cwd=/mock-checkout
Running JavaScriptCore tests
MOCK run_and_throw_if_fail: ['Tools/Scripts/run-javascriptcore-tests', '--no-fail-fast', '--json-output=WebKitBuild/Results/jsc_test_results.json'], cwd=/mock-checkout
Running bindings generation tests
MOCK run_and_throw_if_fail: ['Tools/Scripts/run-bindings-tests'], cwd=/mock-checkout
Running run-webkit-tests

        OutputCapture().assert_outputs(self, step.run, [{}], expected_logs=expected_logs)

    def test_runtests_debug_args(self):
        self.maxDiff = None
        mock_options = self._step_options()
        mock_options.non_interactive = False
        mock_options.build_style = "debug"

Running Perl unit tests
MOCK run_and_throw_if_fail: ['Tools/Scripts/test-webkitperl'], cwd=/mock-checkout
Running JavaScriptCore tests
MOCK run_and_throw_if_fail: ['Tools/Scripts/run-javascriptcore-tests', '--no-fail-fast', '--json-output=WebKitBuild/Results/jsc_test_results.json'], cwd=/mock-checkout
Running bindings generation tests
MOCK run_and_throw_if_fail: ['Tools/Scripts/run-bindings-tests'], cwd=/mock-checkout
Running run-webkit-tests

Return to Bug 162458

Lines 1-3 a/Tools/ChangeLog_sec1
- a/Tools/ChangeLog +41 lines
		1	2016-12-15 Srinivasan Vijayaraghavan <svijayaraghavan@apple.com>
		2
		3	EWS should run JavaScriptCore tests
		4	https://bugs.webkit.org/show_bug.cgi?id=162458
		5
		6	Reviewed by NOBODY (OOPS!).
		7
		8	* Scripts/webkitpy/common/config/ews.json: Add config info for JSC EWS.
		9	* Scripts/webkitpy/common/config/ports.py:
		10	(DeprecatedPort.build_jsc_command): Added - command to build JSC only (quicker than building all of WebKit).
		11	(DeprecatedPort):
		12	(DeprecatedPort.run_javascriptcore_tests_command): Allow JSC EWS to only run JSC tests.
		13	(MacPort.run_webkit_tests_command):
		14	* Scripts/webkitpy/tool/bot/patchanalysistask.py:
		15	(PatchAnalysisTask._build): Check for JSC.
		16	(PatchAnalysisTask._build_without_patch): Check for JSC.
		17	(PatchAnalysisTask._test): Check for JSC.
		18	(PatchAnalysisTask._build_and_test_without_patch): Check for JSC.
		19	(PatchAnalysisTask._test_patch): Add retry logic for JSC.
		20	* Scripts/webkitpy/tool/commands/earlywarningsystem.py:
		21	(AbstractEarlyWarningSystem.begin_work_queue): Use JSCTestResultsReader instead of LayoutTestResultsReader in JSC EWS.
		22	(AbstractEarlyWarningSystem.test_results): _layout_test_results_reader -> _test_results_reader.
		23	(AbstractEarlyWarningSystem.archive_last_test_results): _layout_test_results_reader -> _test_results_reader.
		24	(AbstractEarlyWarningSystem.group): This attribute determines the type of EWS (eg. JSC).
		25	(AbstractEarlyWarningSystem.load_ews_classes): Add _group.
		26	* Scripts/webkitpy/tool/commands/earlywarningsystem_unittest.py:
		27	(EarlyWarningSystemTest._default_expected_logs): Add _group.
		28	* Scripts/webkitpy/tool/steps/build.py:
		29	(Build.options): Check for JSC.
		30	(Build.build): Check for JSC.
		31	(Build.run): Check for JSC.
		32	* Scripts/webkitpy/tool/steps/options.py:
		33	(Options): Add --group command line option.
		34	* Scripts/webkitpy/tool/steps/runtests.py:
		35	(RunTests.options):
		36	(RunTests.run): Check for JSC.
		37	(RunTests._run_webkit_tests): Check for JSC.
		38	(RunTests): Add _group attribute.
		39	(RunTests._run_javascriptcore_tests): New.
		40	* Scripts/webkitpy/tool/steps/steps_unittest.py: Change expected command for running jsc tests.
		41
1	2016-09-25 Gyuyoung Kim <gyuyoung.kim@webkit.org>	42	2016-09-25 Gyuyoung Kim <gyuyoung.kim@webkit.org>
2		43
3	[EFL] Adjust EFL indentation to EFL MiniBrowser	44	[EFL] Adjust EFL indentation to EFL MiniBrowser

Lines 103-110 class DeprecatedPort(object): a/Tools/Scripts/webkitpy/common/config/ports.py_sec1
- a/Tools/Scripts/webkitpy/common/config/ports.py -3 / +19 lines
103	command.append("--release")	103	command.append("--release")
104	return command	104	return command
105		105
106	def run_javascriptcore_tests_command(self):	106	def build_jsc_command(self, build_style=None):
107	return self.script_shell_command("run-javascriptcore-tests")	107	command = self.script_shell_command("build-jsc")
		108	if build_style == "debug":
		109	command.append("--debug")
		110	return command
		111
		112	def run_javascriptcore_tests_command(self, build_style=None):
		113	command = self.script_shell_command("run-javascriptcore-tests")
		114	command.append("--no-fail-fast")
		115	command.append("--json-output=jsc_test_results.json")
		116	command.append("--no-mozilla-tests")
		117	command.append("--no-build")
		118	if (build_style == "debug"):
		119	command.append("--debug")
		120	return command
108		121
109	def run_webkit_tests_command(self, build_style=None):	122	def run_webkit_tests_command(self, build_style=None):
110	command = self.script_shell_command("run-webkit-tests")	123	command = self.script_shell_command("run-webkit-tests")
Lines 150-156 class IOSSimulatorWK2Port(DeprecatedPort): a/Tools/Scripts/webkitpy/common/config/ports.py_sec2
150	class MacPort(DeprecatedPort):	163	class MacPort(DeprecatedPort):
151	port_flag_name = "mac"	164	port_flag_name = "mac"
152		165
153	def run_webkit_tests_command(self, build_style=None):	166	def run_webkit_tests_command(self, build_style=None, test_suite=None):
		167	if (test_suite == "jsc"):
		168	return super(MacPort, self).run_javascriptcore_tests_command(build_style)
		169
154	command = super(MacPort, self).run_webkit_tests_command(build_style)	170	command = super(MacPort, self).run_webkit_tests_command(build_style)
155	command.append("--dump-render-tree")	171	command.append("--dump-render-tree")
156	return command	172	return command

Line 0 a/Tools/Scripts/webkitpy/common/net/jsctestresults.py_sec1
- a/Tools/Scripts/webkitpy/common/net/jsctestresults.py +66 lines
		1	# Copyright (C) 2016 Apple Inc. All rights reserved.
		2	#
		3	# Redistribution and use in source and binary forms, with or without
		4	# modification, are permitted provided that the following conditions
		5	# are met:
		6	# 1. Redistributions of source code must retain the above copyright
		7	# notice, this list of conditions and the following disclaimer.
		8	# 2. Redistributions in binary form must reproduce the above copyright
		9	# notice, this list of conditions and the following disclaimer in the
		10	# documentation and/or other materials provided with the distribution.
		11	#
		12	# THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS'' AND
		13	# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
		14	# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
		15	# DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS BE LIABLE FOR
		16	# ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
		17	# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
		18	# SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
		19	# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
		20	# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
		21	# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
		22
		23	import json
		24	import logging
		25	from webkitpy.thirdparty.BeautifulSoup import BeautifulSoup, SoupStrainer
		26
		27	_log = logging.getLogger(__name__)
		28
		29
		30	class JSCTestResults(object):
		31	@classmethod
		32	def results_from_string(cls, string):
		33	if not string:
		34	return None
		35	parsed_results = json.load(string)
		36	return cls(parsed_results['allApiTestsPassed'], parsed_results['stressTestFailures'])
		37
		38	def __init__(self, all_api_tests_passed, stress_test_failures):
		39	self._all_api_tests_passed = all_api_tests_passed
		40	self._stress_test_failures = stress_test_failures
		41
		42	self._test_results = stress_test_failures
		43	if self._all_api_tests_passed == False:
		44	self._test_results += "apiTests"
		45
		46	def stress_test_failures(self):
		47	return self._stress_test_failures
		48
		49	def all_api_tests_passed(self):
		50	return self._all_api_tests_passed
		51
		52	def all_passed(self):
		53	return self._all_api_tests_passed == True and len(self._stress_test_failures) == 0
		54
		55	def failing_test_results(self):
		56	return self.stress_test_failures()
		57
		58	def test_results(self):
		59	return self._test_results
		60
		61	def failing_tests(self):
		62	return self._test_results
		63
		64	# No defined failure limit for JSC
65	def did_exceed_test_failure_limit(self):
66	return False

Line 0 a/Tools/Scripts/webkitpy/tool/bot/jsctestresultsreader.py_sec1
- a/Tools/Scripts/webkitpy/tool/bot/jsctestresultsreader.py +47 lines
	1	# Copyright (C) 2016 Apple Inc. All rights reserved.
	2	#
	3	# Redistribution and use in source and binary forms, with or without
	4	# modification, are permitted provided that the following conditions
	5	# are met:
	6	# 1. Redistributions of source code must retain the above copyright
	7	# notice, this list of conditions and the following disclaimer.
	8	# 2. Redistributions in binary form must reproduce the above copyright
	9	# notice, this list of conditions and the following disclaimer in the
	10	# documentation and/or other materials provided with the distribution.
	11	#
	12	# THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS'' AND
	13	# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
	14	# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
	15	# DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS BE LIABLE FOR
	16	# ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
	17	# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
	18	# SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
	19	# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
	20	# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
	21	# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
	22
	23	import logging
	24	import json
	25	from webkitpy.common.net.jsctestresults import JSCTestResults
	26
	27	_log = logging.getLogger(__name__)
	28
	29
	30	class JSCTestResultsReader(object):
	31	def __init__(self, host, results_directory):
	32	self._host = host
	33	self._results_directory = results_directory
	34
	35	def _read_file_contents(self, path):
	36	try:
	37	return self._host.filesystem.read_text_file(path)
	38	except IOError, e: # File does not exist or can't be read.
	39	return None
	40
	41	def _create_jsc_test_results(self):
	42	results_path = self._host.filesystem.join(self._results_directory, "jsc_test_results.json")
	43	contents = open(results_path)
	44	return JSCTestResults.results_from_string(contents)
	45
	46	def results(self):
	47	return self._create_jsc_test_results()

Lines 123-149 class PatchAnalysisTask(object): a/Tools/Scripts/webkitpy/tool/bot/patchanalysistask.py_sec1
- a/Tools/Scripts/webkitpy/tool/bot/patchanalysistask.py -65 / +108 lines
123	"Patch does not apply")	123	"Patch does not apply")
124		124
125	def _build(self):	125	def _build(self):
126	return self._run_command([	126	args = [
127	"build",	127	"build",
128	"--no-clean",	128	"--no-clean",
129	"--no-update",	129	"--no-update",
130	"--build-style=%s" % self._delegate.build_style(),	130	"--build-style=%s" % self._delegate.build_style(),
131	],	131	]
132	"Built patch",	132
133	"Patch does not build")	133	try:
		134	args.append("--group=%s" % self._delegate.group())
		135	finally:
		136	return self._run_command(
		137	args,
		138	"Built patch",
		139	"Patch does not build")
134		140
135	def _build_without_patch(self):	141	def _build_without_patch(self):
136	return self._run_command([	142	args = [
137	"build",	143	"build",
138	"--force-clean",	144	"--force-clean",
139	"--no-update",	145	"--no-update",
140	"--build-style=%s" % self._delegate.build_style(),	146	"--build-style=%s" % self._delegate.build_style(),
141	],	147	]
142	"Able to build without patch",	148
143	"Unable to build without patch")	149	try:
		150	args.append("--group=%s" % self._delegate.group())
		151	finally:
		152	return self._run_command(
		153	args,
		154	"Able to build without patch",
		155	"Unable to build without patch")
144		156
145	def _test(self):	157	def _test(self):
146	return self._run_command([	158	args = [
147	"build-and-test",	159	"build-and-test",
148	"--no-clean",	160	"--no-clean",
149	"--no-update",	161	"--no-update",
Lines 151-162 class PatchAnalysisTask(object): a/Tools/Scripts/webkitpy/tool/bot/patchanalysistask.py_sec2
151	"--test",	163	"--test",
152	"--non-interactive",	164	"--non-interactive",
153	"--build-style=%s" % self._delegate.build_style(),	165	"--build-style=%s" % self._delegate.build_style(),
154	],	166	]
155	"Passed tests",	167
156	"Patch does not pass tests")	168	try:
		169	args.append("--group=%s" % self._delegate.group())
		170	finally:
		171	return self._run_command(
		172	args,
		173	"Passed tests",
		174	"Patch does not pass tests")
157		175
158	def _build_and_test_without_patch(self):	176	def _build_and_test_without_patch(self):
159	return self._run_command([	177	args = [
160	"build-and-test",	178	"build-and-test",
161	"--force-clean",	179	"--force-clean",
162	"--no-update",	180	"--no-update",
Lines 164-172 class PatchAnalysisTask(object): a/Tools/Scripts/webkitpy/tool/bot/patchanalysistask.py_sec3
164	"--test",	182	"--test",
165	"--non-interactive",	183	"--non-interactive",
166	"--build-style=%s" % self._delegate.build_style(),	184	"--build-style=%s" % self._delegate.build_style(),
167	],	185	]
168	"Able to pass tests without patch",	186
169	"Unable to pass tests without patch (tree is red?)")	187	try:
		188	args.append("--group=%s" % self._delegate.group())
		189	finally:
		190	return self._run_command(
		191	args,
		192	"Able to pass tests without patch",
		193	"Unable to pass tests without patch (tree is red?)")
170		194
171	def _land(self):	195	def _land(self):
172	# Unclear if this should pass --quiet or not. If --parent-command always does the reporting, then it should.	196	# Unclear if this should pass --quiet or not. If --parent-command always does the reporting, then it should.
Lines 210-278 class PatchAnalysisTask(object): a/Tools/Scripts/webkitpy/tool/bot/patchanalysistask.py_sec4
210	if self._test():	234	if self._test():
211	return True	235	return True
212		236
213	# Note: archive_last_test_results deletes the results directory, making these calls order-sensitve.	237	if hasattr(self._delegate, 'group') and self._delegate.group() == "jsc":
214	# We could remove this dependency by building the test_results from the archive.	238	first_results = self._delegate.test_results()
215	first_results = self._delegate.test_results()	239	first_script_error = self._script_error
216	first_results_archive = self._delegate.archive_last_test_results(self._patch)	240	if first_results.all_api_tests_passed() is True and len(first_results.stress_test_failures()) == 0:
217	first_script_error = self._script_error	241	return True
218	first_failure_status_id = self.failure_status_id	242	else:
219		243	# TODO svijayaraghavan@apple.com: Comment this, and uncomment the below lines so it actually builds without patch
220	if self._test() and not first_results.did_exceed_test_failure_limit():	244	#clean_tree_results = first_results
221	# Only report flaky tests if we were successful at parsing results.json and archiving results.	245	#clean_tree_results['allApiTestsPassed'] = False
222	if first_results and first_results_archive:
223	self._report_flaky_tests(first_results.failing_test_results(), first_results_archive)
224	return True
225		246
226	second_results = self._delegate.test_results()	247	self._build_and_test_without_patch()
227	second_results_archive = self._delegate.archive_last_test_results(self._patch)	248	clean_tree_results = self._delegate.test_results()
228	second_script_error = self._script_error
229	second_failure_status_id = self.failure_status_id
230		249
231	if second_results.did_exceed_test_failure_limit() and first_results.did_exceed_test_failure_limit():	250	if clean_tree_results == first_results:
232	self._build_and_test_without_patch()	251	return True
233	clean_tree_results = self._delegate.test_results()
234		252
235	if (len(first_results.failing_tests()) - len(clean_tree_results.failing_tests())) <= 5:	253	return self.report_failure(None, first_results, first_script_error)
236	return False
237		254
238	self.failure_status_id = first_failure_status_id	255	else:
		256	# Note: archive_last_test_results deletes the results directory, making these calls order-sensitve.
		257	# We could remove this dependency by building the test_results from the archive.
		258	first_results = self._delegate.test_results()
		259	first_results_archive = self._delegate.archive_last_test_results(self._patch)
		260	first_script_error = self._script_error
		261	first_failure_status_id = self.failure_status_id
239		262
240	return self.report_failure(first_results_archive, first_results, first_script_error)	263	if self._test() and not first_results.did_exceed_test_failure_limit():
		264	# Only report flaky tests if we were successful at parsing results.json and archiving results.
		265	if first_results and first_results_archive:
		266	self._report_flaky_tests(first_results.failing_test_results(), first_results_archive)
		267	return True
241		268
242	if second_results.did_exceed_test_failure_limit():	269	second_results = self._delegate.test_results()
243	self._should_defer_patch_or_throw(first_results.failing_test_results(), first_results_archive, first_script_error, first_failure_status_id)	270	second_results_archive = self._delegate.archive_last_test_results(self._patch)
244	return False	271	second_script_error = self._script_error
		272	second_failure_status_id = self.failure_status_id
245		273
246	if first_results.did_exceed_test_failure_limit():	274	if second_results.did_exceed_test_failure_limit() and first_results.did_exceed_test_failure_limit():
247	self._should_defer_patch_or_throw(second_results.failing_test_results(), second_results_archive, second_script_error, second_failure_status_id)	275	self._build_and_test_without_patch()
248	return False	276	clean_tree_results = self._delegate.test_results()
		277
		278	if (len(first_results.failing_tests()) - len(clean_tree_results.failing_tests())) <= 5:
		279	return False
249		280
250	if self._results_failed_different_tests(first_results, second_results):	281	self.failure_status_id = first_failure_status_id
251	first_failing_results_set = frozenset(first_results.failing_test_results())
252	second_failing_results_set = frozenset(second_results.failing_test_results())
253		282
254	tests_that_only_failed_first = first_failing_results_set.difference(second_failing_results_set)	283	return self.report_failure(first_results_archive, first_results, first_script_error)
255	self._report_flaky_tests(tests_that_only_failed_first, first_results_archive)
256		284
257	tests_that_only_failed_second = second_failing_results_set.difference(first_failing_results_set)	285	if second_results.did_exceed_test_failure_limit():
258	self._report_flaky_tests(tests_that_only_failed_second, second_results_archive)	286	self._should_defer_patch_or_throw(first_results.failing_test_results(), first_results_archive, first_script_error, first_failure_status_id)
		287	return False
		288
		289	if first_results.did_exceed_test_failure_limit():
		290	self._should_defer_patch_or_throw(second_results.failing_test_results(), second_results_archive, second_script_error, second_failure_status_id)
		291	return False
		292
		293	if self._results_failed_different_tests(first_results, second_results):
		294	first_failing_results_set = frozenset(first_results.failing_test_results())
		295	second_failing_results_set = frozenset(second_results.failing_test_results())
259		296
260	tests_that_consistently_failed = first_failing_results_set.intersection(second_failing_results_set)	297	tests_that_only_failed_first = first_failing_results_set.difference(second_failing_results_set)
261	if tests_that_consistently_failed:	298	self._report_flaky_tests(tests_that_only_failed_first, first_results_archive)
262	if self._should_defer_patch_or_throw(tests_that_consistently_failed, first_results_archive, first_script_error, first_failure_status_id):
263	return False # Defer patch
264		299
265	# At this point we know that at least one test flaked, but no consistent failures	300	tests_that_only_failed_second = second_failing_results_set.difference(first_failing_results_set)
266	# were introduced. This is a bit of a grey-zone.	301	self._report_flaky_tests(tests_that_only_failed_second, second_results_archive)
267	return False # Defer patch
268		302
269	if self._should_defer_patch_or_throw(first_results.failing_test_results(), first_results_archive, first_script_error, first_failure_status_id):	303	tests_that_consistently_failed = first_failing_results_set.intersection(second_failing_results_set)
270	return False # Defer patch	304	if tests_that_consistently_failed:
		305	if self._should_defer_patch_or_throw(tests_that_consistently_failed, first_results_archive, first_script_error, first_failure_status_id):
		306	return False # Defer patch
271		307
272	# At this point, we know that the first and second runs had the exact same failures,	308	# At this point we know that at least one test flaked, but no consistent failures
273	# and that those failures are all present on the clean tree, so we can say with certainty	309	# were introduced. This is a bit of a grey-zone.
274	# that the patch is good.	310	return False # Defer patch
275	return True	311
		312	if self._should_defer_patch_or_throw(first_results.failing_test_results(), first_results_archive, first_script_error, first_failure_status_id):
		313	return False # Defer patch
		314
		315	# At this point, we know that the first and second runs had the exact same failures,
		316	# and that those failures are all present on the clean tree, so we can say with certainty
		317	# that the patch is good.
		318	return True
276		319
277	def results_archive_from_patch_test_run(self, patch):	320	def results_archive_from_patch_test_run(self, patch):
278	assert(self._patch.id() == patch.id()) # PatchAnalysisTask is not currently re-useable.	321	assert(self._patch.id() == patch.id()) # PatchAnalysisTask is not currently re-useable.

Lines 68-76 class EarlyWarningSystemTest(QueuesTest): a/Tools/Scripts/webkitpy/tool/commands/earlywarningsystem_unittest.py_sec1
- a/Tools/Scripts/webkitpy/tool/commands/earlywarningsystem_unittest.py -2 / +3 lines
68	"port": ews.port_name,	68	"port": ews.port_name,
69	"architecture": " --architecture=%s" % ews.architecture if ews.architecture else "",	69	"architecture": " --architecture=%s" % ews.architecture if ews.architecture else "",
70	"build_style": ews.build_style(),	70	"build_style": ews.build_style(),
		71	"group": "jsc" if ews.name == "jsc-ews" else "None",
71	}	72	}
72	if ews.run_tests:	73	if ews.run_tests:
73	run_tests_line = "Running: webkit-patch --status-host=example.com build-and-test --no-clean --no-update --test --non-interactive --build-style=%(build_style)s --port=%(port)s%(architecture)s\n" % string_replacements	74	run_tests_line = "Running: webkit-patch --status-host=example.com build-and-test --no-clean --no-update --test --non-interactive --build-style=%(build_style)s --group=%(group)s --port=%(port)s%(architecture)s\n" % string_replacements
74	else:	75	else:
75	run_tests_line = ""	76	run_tests_line = ""
76	string_replacements['run_tests_line'] = run_tests_line	77	string_replacements['run_tests_line'] = run_tests_line
Lines 81-87 class EarlyWarningSystemTest(QueuesTest): a/Tools/Scripts/webkitpy/tool/commands/earlywarningsystem_unittest.py_sec2
81	Running: webkit-patch --status-host=example.com clean --port=%(port)s%(architecture)s	82	Running: webkit-patch --status-host=example.com clean --port=%(port)s%(architecture)s
82	Running: webkit-patch --status-host=example.com update --port=%(port)s%(architecture)s	83	Running: webkit-patch --status-host=example.com update --port=%(port)s%(architecture)s
83	Running: webkit-patch --status-host=example.com apply-attachment --no-update --non-interactive 10000 --port=%(port)s%(architecture)s	84	Running: webkit-patch --status-host=example.com apply-attachment --no-update --non-interactive 10000 --port=%(port)s%(architecture)s
84	Running: webkit-patch --status-host=example.com build --no-clean --no-update --build-style=%(build_style)s --port=%(port)s%(architecture)s	85	Running: webkit-patch --status-host=example.com build --no-clean --no-update --build-style=%(build_style)s --group=%(group)s --port=%(port)s%(architecture)s
85	%(run_tests_line)sMOCK: update_status: %(name)s Pass	86	%(run_tests_line)sMOCK: update_status: %(name)s Pass
86	MOCK: release_work_item: %(name)s 10000	87	MOCK: release_work_item: %(name)s 10000
87	""" % string_replacements,	88	""" % string_replacements,

Lines 41-68 class Build(AbstractStep): a/Tools/Scripts/webkitpy/tool/steps/build.py_sec1
- a/Tools/Scripts/webkitpy/tool/steps/build.py -7 / +17 lines
41	Options.build,	41	Options.build,
42	Options.quiet,	42	Options.quiet,
43	Options.build_style,	43	Options.build_style,
		44	Options.group,
44	]	45	]
45		46
46	def build(self, build_style):	47	def build(self, build_style, group):
47	environment = self._tool.copy_current_environment()	48	environment = self._tool.copy_current_environment()
48	environment.disable_gcc_smartquotes()	49	environment.disable_gcc_smartquotes()
49	environment.disable_jhbuild_VT100_output()	50	environment.disable_jhbuild_VT100_output()
50	env = environment.to_dictionary()	51	env = environment.to_dictionary()
51		52
52	build_webkit_command = self._tool.deprecated_port().build_webkit_command(build_style=build_style)	53	if group == "jsc":
		54	build_command = self._tool.deprecated_port().build_jsc_command(build_style=build_style)
		55	else:
		56	build_command = self._tool.deprecated_port().build_webkit_command(build_style=build_style)
53		57
54	if self._options.architecture:	58	if self._options.architecture:
55	build_webkit_command += ['ARCHS=%s' % self._options.architecture]	59	build_command += ['ARCHS=%s' % self._options.architecture]
56		60
57	self._tool.executive.run_and_throw_if_fail(build_webkit_command, self._options.quiet,	61	self._tool.executive.run_and_throw_if_fail(build_command, self._options.quiet,
58	cwd=self._tool.scm().checkout_root, env=env)	62	cwd=self._tool.scm().checkout_root, env=env)
59		63
60	def run(self, state):	64	def run(self, state):
61	if not self._options.build:	65	if not self._options.build:
62	return	66	return
63	_log.info("Building WebKit")	67	_log.info("Building WebKit")
		68
		69	if hasattr(self._options, 'group') and self._options.group == "jsc":
		70	group = self._options.group
		71	else:
		72	group = None
		73
64	if self._options.build_style == "both":	74	if self._options.build_style == "both":
65	self.build("debug")	75	self.build("debug", group)
66	self.build("release")	76	self.build("release", group)
67	else:	77	else:
68	self.build(self._options.build_style)	78	self.build(self._options.build_style, group)

Lines 98-103 class StepsTest(unittest.TestCase): a/Tools/Scripts/webkitpy/tool/steps/steps_unittest.py_sec1
- a/Tools/Scripts/webkitpy/tool/steps/steps_unittest.py -2 / +4 lines
98	self._assert_step_output_with_bug(steps.EnsureBugIsOpenAndAssigned, 50002, expected_logs)	98	self._assert_step_output_with_bug(steps.EnsureBugIsOpenAndAssigned, 50002, expected_logs)
99		99
100	def test_runtests_args(self):	100	def test_runtests_args(self):
		101	self.maxDiff = None
101	mock_options = self._step_options()	102	mock_options = self._step_options()
102	mock_options.non_interactive = False	103	mock_options.non_interactive = False
103	mock_options.build_style = "release"	104	mock_options.build_style = "release"
Lines 111-117 MOCK run_and_throw_if_fail: ['Tools/Scripts/test-webkitpy'], cwd=/mock-checkout a/Tools/Scripts/webkitpy/tool/steps/steps_unittest.py_sec2
111	Running Perl unit tests	112	Running Perl unit tests
112	MOCK run_and_throw_if_fail: ['Tools/Scripts/test-webkitperl'], cwd=/mock-checkout	113	MOCK run_and_throw_if_fail: ['Tools/Scripts/test-webkitperl'], cwd=/mock-checkout
113	Running JavaScriptCore tests	114	Running JavaScriptCore tests
114	MOCK run_and_throw_if_fail: ['Tools/Scripts/run-javascriptcore-tests'], cwd=/mock-checkout	115	MOCK run_and_throw_if_fail: ['Tools/Scripts/run-javascriptcore-tests', '--no-fail-fast', '--json-output=WebKitBuild/Results/jsc_test_results.json'], cwd=/mock-checkout
115	Running bindings generation tests	116	Running bindings generation tests
116	MOCK run_and_throw_if_fail: ['Tools/Scripts/run-bindings-tests'], cwd=/mock-checkout	117	MOCK run_and_throw_if_fail: ['Tools/Scripts/run-bindings-tests'], cwd=/mock-checkout
117	Running run-webkit-tests	118	Running run-webkit-tests
Lines 120-125 MOCK run_and_throw_if_fail: ['Tools/Scripts/run-webkit-tests', '--release', '--q a/Tools/Scripts/webkitpy/tool/steps/steps_unittest.py_sec3
120	OutputCapture().assert_outputs(self, step.run, [{}], expected_logs=expected_logs)	121	OutputCapture().assert_outputs(self, step.run, [{}], expected_logs=expected_logs)
121		122
122	def test_runtests_debug_args(self):	123	def test_runtests_debug_args(self):
		124	self.maxDiff = None
123	mock_options = self._step_options()	125	mock_options = self._step_options()
124	mock_options.non_interactive = False	126	mock_options.non_interactive = False
125	mock_options.build_style = "debug"	127	mock_options.build_style = "debug"
Lines 133-139 MOCK run_and_throw_if_fail: ['Tools/Scripts/test-webkitpy'], cwd=/mock-checkout a/Tools/Scripts/webkitpy/tool/steps/steps_unittest.py_sec4
133	Running Perl unit tests	135	Running Perl unit tests
134	MOCK run_and_throw_if_fail: ['Tools/Scripts/test-webkitperl'], cwd=/mock-checkout	136	MOCK run_and_throw_if_fail: ['Tools/Scripts/test-webkitperl'], cwd=/mock-checkout
135	Running JavaScriptCore tests	137	Running JavaScriptCore tests
136	MOCK run_and_throw_if_fail: ['Tools/Scripts/run-javascriptcore-tests'], cwd=/mock-checkout	138	MOCK run_and_throw_if_fail: ['Tools/Scripts/run-javascriptcore-tests', '--no-fail-fast', '--json-output=WebKitBuild/Results/jsc_test_results.json'], cwd=/mock-checkout
137	Running bindings generation tests	139	Running bindings generation tests
138	MOCK run_and_throw_if_fail: ['Tools/Scripts/run-bindings-tests'], cwd=/mock-checkout	140	MOCK run_and_throw_if_fail: ['Tools/Scripts/run-bindings-tests'], cwd=/mock-checkout
139	Running run-webkit-tests	141	Running run-webkit-tests