Attachment 346782 Details for Bug 188413 – Patch

[patch] Patch

bug-188413-20180808121041.patch (text/plain), 57.15 KB, created by Jonathan Bedard on 2018-08-08 12:10:42 PDT

(hide)

Description:

Filename:

MIME Type:

Creator: Jonathan Bedard

Created: 2018-08-08 12:10:42 PDT

Size: 57.15 KB

patch

obsolete

>Index: Tools/ChangeLog
>===================================================================
>--- Tools/ChangeLog	(revision 234703)
>+++ Tools/ChangeLog	(working copy)
>@@ -1,3 +1,47 @@
>+2018-08-08  Jonathan Bedard  <jbedard@apple.com>
>+
>+        Add basic flakiness-analyzer
>+        https://bugs.webkit.org/show_bug.cgi?id=188413
>+
>+        Reviewed by NOBODY (OOPS!).
>+
>+        Add a flakiness-analyzer script which uses the stdout of a test run to generate
>+        a test order which may reproduce the test failure.
>+
>+        * Scripts/flakiness-analyzer: Added.
>+        (parse_args):
>+        * Scripts/webkitpy/flakiness_analyzer: Added.
>+        * Scripts/webkitpy/flakiness_analyzer/__init__.py: Added.
>+        * Scripts/webkitpy/flakiness_analyzer/flakiness_analyzer.py: Added.
>+        (determine_focus): If no focus was provided, treat the first unexpected failure as
>+        the focus.
>+        (processes_for_focus): Return all instances of a Process which contain a test run
>+        matching the specified focus.
>+        (potential_reproduction_steps_for_processes_with_focus): Given a process, return a list
>+        of tests which could possibly reproduce the failure seen in the provided process.
>+        (combine_similar_reproduction_steps): If one sequence of reproduction steps contains another,
>+        the shorter sequence is more useful.
>+        (analyze_stdouts): Given path to stdouts, print out potential reproduction steps.
>+        * Scripts/webkitpy/flakiness_analyzer/flakiness_analyzer_unittest.py: Added.
>+        (FlakinessAnalyzerTest):
>+        * Scripts/webkitpy/flakiness_analyzer/test_run.py: Added.
>+        (Test): Abstraction to hold a single Test result.
>+        (TestRun):
>+        (TestRun._find_number_of_workers):
>+        (TestRun._find_next_test):
>+        (TestRun.from_stdout): Parses the verbose stdout off a test run and reconstructs the order
>+        tests were ran in.
>+        (TestRun.Process): Holds the order tests were ran in a single instance of the test runner,
>+        because these test results will be related.
>+        (TestRun.Process.from_test_list): Given a list of tests run in one of run-webkit-tests'
>+        shards, separate those test runs into processes based on where the TestRunner was forced
>+        to restart.
>+        (TestRun.Process.__init__):
>+        (TestRun.Process.contains_test): Check if a process contains a specific focus.
>+        (TestRun.Process.failed_tests): Return all tests which failed unexpectedly.
>+        (TestRun.__init__):
>+        (TestRun.processes_with_tests): 
>+
> 2018-08-08  Wenson Hsieh  <wenson_hsieh@apple.com>
> 
>         [iOS] fast/events/ios/contenteditable-autocapitalize.html is a flaky failure
>Index: Tools/Scripts/flakiness-analyzer
>===================================================================
>--- Tools/Scripts/flakiness-analyzer	(nonexistent)
>+++ Tools/Scripts/flakiness-analyzer	(working copy)
>@@ -0,0 +1,65 @@
>+#!/usr/bin/env python
>+
>+# Copyright (C) 2018 Apple Inc. All rights reserved.
>+#
>+# Redistribution and use in source and binary forms, with or without
>+# modification, are permitted provided that the following conditions
>+# are met:
>+# 1.  Redistributions of source code must retain the above copyright
>+#    notice, this list of conditions and the following disclaimer.
>+# 2.  Redistributions in binary form must reproduce the above copyright
>+#    notice, this list of conditions and the following disclaimer in the
>+#    documentation and/or other materials provided with the distribution.
>+#
>+# THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS'' AND ANY
>+# EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
>+# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
>+# DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS BE LIABLE FOR ANY
>+# DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
>+# (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
>+# LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
>+# ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
>+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
>+# SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
>+
>+import optparse
>+import logging
>+import sys
>+
>+from webkitpy.flakiness_analyzer.flakiness_analyzer import analyze_stdouts
>+
>+def parse_args(args):
>+    option_group_definitions = []
>+
>+    option_group_definitions.append(('Focus Options', [
>+        optparse.make_option('-f', '--focus', action='store', default=None,
>+                             help='Test or test suite to focus on. By default, this will be the first failing (or crashing) test found'),
>+    ]))
>+    option_group_definitions.append(('General Options', [
>+        optparse.make_option('-v', '--verbose', action='store_true', default=False,
>+                             help='Enable verbose logging'),
>+    ]))
>+
>+    option_parser = optparse.OptionParser(
>+        usage='%prog [options] [<test stdout paths>...]',
>+        description="""This program attempts to output a list of tests which can reproduce a specific failure. In-order
>+to do this, this program requires verbose stdout from a run-webkit-tests run. If multiple stdouts are provided, their
>+outputs will be combined to generate the shortest possible sequence of tests required to reproduce the problem.
>+"""
>+    )
>+
>+    for group_name, group_options in option_group_definitions:
>+        option_group = optparse.OptionGroup(option_parser, group_name)
>+        option_group.add_options(group_options)
>+        option_parser.add_option_group(option_group)
>+
>+    return option_parser.parse_args(args)
>+
>+
>+if __name__ == '__main__':
>+    logging.basicConfig(level=logging.WARNING)
>+    options, args = parse_args(sys.argv[1:])
>+
>+    if options.verbose:
>+        logging.getLogger().setLevel(logging.INFO)
>+    sys.exit(analyze_stdouts(args, focus=options.focus))
>
>Property changes on: Tools/Scripts/flakiness-analyzer
>___________________________________________________________________
>Added: svn:executable
>## -0,0 +1 ##
>+*
>\ No newline at end of property
>Index: Tools/Scripts/webkitpy/flakiness_analyzer/__init__.py
>===================================================================
>--- Tools/Scripts/webkitpy/flakiness_analyzer/__init__.py	(nonexistent)
>+++ Tools/Scripts/webkitpy/flakiness_analyzer/__init__.py	(working copy)
>@@ -0,0 +1,13 @@
>+# Required for Python to search this directory for module files
>+
>+# Keep this file free of any code or import statements that could
>+# cause either an error to occur or a log message to be logged.
>+# This ensures that calling code can import initialization code from
>+# webkitpy before any errors or log messages due to code in this file.
>+# Initialization code can include things like version-checking code and
>+# logging configuration code.
>+#
>+# We do not execute any version-checking code or logging configuration
>+# code in this file so that callers can opt-in as they want.  This also
>+# allows different callers to choose different initialization code,
>+# as necessary.
>Index: Tools/Scripts/webkitpy/flakiness_analyzer/flakiness_analyzer.py
>===================================================================
>--- Tools/Scripts/webkitpy/flakiness_analyzer/flakiness_analyzer.py	(nonexistent)
>+++ Tools/Scripts/webkitpy/flakiness_analyzer/flakiness_analyzer.py	(working copy)
>@@ -0,0 +1,160 @@
>+# Copyright (C) 2018 Apple Inc. All rights reserved.
>+#
>+# Redistribution and use in source and binary forms, with or without
>+# modification, are permitted provided that the following conditions
>+# are met:
>+# 1.  Redistributions of source code must retain the above copyright
>+#    notice, this list of conditions and the following disclaimer.
>+# 2.  Redistributions in binary form must reproduce the above copyright
>+#    notice, this list of conditions and the following disclaimer in the
>+#    documentation and/or other materials provided with the distribution.
>+#
>+# THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS'' AND ANY
>+# EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
>+# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
>+# DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS BE LIABLE FOR ANY
>+# DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
>+# (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
>+# LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
>+# ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
>+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
>+# SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
>+
>+from webkitpy.common.host import Host
>+from webkitpy.flakiness_analyzer.test_run import TestRun
>+
>+import logging
>+
>+
>+log = logging.getLogger(__name__)
>+
>+
>+def determine_focus(focus, test_runs):
>+    if focus:
>+        return focus
>+    for run in test_runs:
>+        for process in run.processes:
>+            for test in process.tests:
>+                if test.result != 'passed' and not test.expected:
>+                    return test.name
>+    return focus
>+
>+
>+def processes_for_focus(focus, test_runs):
>+    result = []
>+    for run in test_runs:
>+        relevant_processes_for_focus = run.processes_with_tests([focus])
>+        relevant_test_always_first = True
>+        for process in relevant_processes_for_focus:
>+            if not process.tests[0].name.startswith(focus):
>+                result.append(process)
>+                relevant_test_always_first = False
>+        if relevant_test_always_first:
>+            result += relevant_processes_for_focus
>+    return result
>+
>+
>+def potential_reproduction_steps_for_processes_with_focus(failed_process, successful_process_list, focus):
>+    result = []
>+    failed_tests = failed_process.failed_tests(focus)
>+    for failed_test in failed_tests:
>+        result.append([])
>+        for test in failed_process.tests:
>+            result[-1].append(test.name)
>+            if test.name == failed_test.name:
>+                break
>+        for process in successful_process_list:
>+            if not process.contains_test([failed_test.name]):
>+                continue
>+            for test in process.tests:
>+                if test.name in result[-1] and test.name != failed_test.name:
>+                    result[-1].remove(test.name)
>+    return combine_similar_reproduction_steps(result)
>+
>+
>+def combine_similar_reproduction_steps(list_of_steps):
>+    if len(list_of_steps) <= 1:
>+        return list_of_steps
>+    if len(list_of_steps) == 2:
>+        index = 0
>+        while index < len(list_of_steps[0]) and index < len(list_of_steps[1]):
>+            if list_of_steps[0][index] != list_of_steps[1][index]:
>+                return list_of_steps
>+            index += 1
>+        if len(list_of_steps[0]) < len(list_of_steps[1]):
>+            [list_of_steps[0]]
>+        return [list_of_steps[1]]
>+
>+    result = list_of_steps
>+    i1 = 0
>+
>+    # We can't use a for loop because result is modified inside the loop
>+    while i1 < len(result):
>+        i2 = i1 + 1
>+        while i2 < len(result):
>+            combination = combine_similar_reproduction_steps([result[i1], result[i2]])
>+            if len(combination) == 1:
>+                result[i1] = combination[0]
>+                del result[i2]
>+            i2 += 1
>+        i1 += 1
>+
>+    return result
>+
>+
>+def analyze_stdouts(stdout_paths, focus=None, host=None):
>+    if not stdout_paths:
>+        log.error('No stdout provided, cannot preform flakiness analysis')
>+        return -1
>+
>+    host = host or Host()
>+
>+    # Grab all process
>+    test_runs = []
>+    log.info('Reading in test stdout...')
>+    for path in stdout_paths:
>+        log.info('    {}'.format(path))
>+        test_runs.append(TestRun.from_stdout(host.filesystem.open_text_file_for_reading(path)))
>+
>+    log.info('Determining which test to focus on...')
>+    focus = determine_focus(focus, test_runs)
>+    log.info('Focusing on {}'.format(focus))
>+    if focus == None:
>+        log.info('A focus is required to analyze flakiness')
>+        return -1
>+
>+    log.info('Finding processes for focus...')
>+    processes = processes_for_focus(focus, test_runs)
>+    log.info('Found {} processes from {} test test runs'.format(len(processes), len(test_runs)))
>+
>+    log.info('Determining which processes have failures...')
>+    successful_examples = []
>+    failed_examples = []
>+    processes_with_failure = 0
>+    for process in processes:
>+        if process.failed_tests([focus]):
>+            processes_with_failure += 1
>+            failed_examples.append(process)
>+        else:
>+            successful_examples.append(process)
>+    log.info('Found {} examples of failed processes and {} examples of successful processes'.format(len(failed_examples), len(successful_examples)))
>+    if processes_with_failure == 0:
>+        log.error('No failures in the provided focus, nothing to analyze')
>+        return -1
>+
>+    log.info('Comparing failed process and successful processes for a common sequence....')
>+    steps_candidates = []
>+    for failed_process in failed_examples:
>+        steps_candidates += potential_reproduction_steps_for_processes_with_focus(failed_process, successful_examples, focus)
>+    steps_candidates = combine_similar_reproduction_steps(list_of_steps=steps_candidates)
>+    log.info('{} sequence(s) that might reproduce the failure'.format(len(steps_candidates)))
>+    index_to_print = 0
>+    if len(steps_candidates) != 1:
>+        index_to_print = int(raw_input('Pick which of the sequences (0-{}) to print: '.format(len(steps_candidates - 1))))
>+        if index_to_print >= len(steps_candidates):
>+            log.error('{} is out of range of the number of step candidates'.format(index_to_print))
>+            return -1
>+    for test in steps_candidates[index_to_print]:
>+        print test
>+
>+    return 0
>Index: Tools/Scripts/webkitpy/flakiness_analyzer/flakiness_analyzer_unittest.py
>===================================================================
>--- Tools/Scripts/webkitpy/flakiness_analyzer/flakiness_analyzer_unittest.py	(nonexistent)
>+++ Tools/Scripts/webkitpy/flakiness_analyzer/flakiness_analyzer_unittest.py	(working copy)
>@@ -0,0 +1,594 @@
>+# Copyright (C) 2018 Apple Inc. All rights reserved.
>+#
>+# Redistribution and use in source and binary forms, with or without
>+# modification, are permitted provided that the following conditions
>+# are met:
>+# 1.  Redistributions of source code must retain the above copyright
>+#    notice, this list of conditions and the following disclaimer.
>+# 2.  Redistributions in binary form must reproduce the above copyright
>+#    notice, this list of conditions and the following disclaimer in the
>+#    documentation and/or other materials provided with the distribution.
>+#
>+# THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS'' AND ANY
>+# EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
>+# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
>+# DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS BE LIABLE FOR ANY
>+# DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
>+# (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
>+# LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
>+# ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
>+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
>+# SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
>+
>+import unittest
>+
>+from webkitpy.common.host_mock import MockHost
>+from webkitpy.common.system.outputcapture import OutputCapture, OutputCaptureScope
>+from webkitpy.flakiness_analyzer.flakiness_analyzer import analyze_stdouts
>+
>+
>+class FlakinessAnalyzerTest(unittest.TestCase):
>+
>+    maxDiff = None
>+    SUCCESS_STDOUT = """14:45:21.473 51856 "sysctl -n hw.memsize" took 0.01s
>+14:45:21.667 51856 "perl Tools/Scripts/webkit-build-directory --configuration --debug --mac" took 0.19s
>+14:45:21.669 51856 Using port 'mac-mojave-wk2'
>+14:45:21.669 51856 Test configuration: <mojave, x86_64, debug>
>+14:45:21.669 51856 Placing test results in /Volumes/Data/Code/Customer/OpenSource/WebKitBuild/Debug/layout-test-results
>+14:45:21.669 51856 Baseline search path: platform/mac-mojave-wk2 -> platform/mac-mojave -> platform/mac-wk2 -> platform/mac -> platform/wk2 -> generic
>+14:45:21.669 51856 Using Debug build
>+14:45:21.669 51856 Pixel tests disabled
>+14:45:21.669 51856 Regular timeout: 30000, slow test timeout: 150000
>+14:45:21.670 51856 Command line: /Volumes/Data/Code/Customer/OpenSource/WebKitBuild/Debug/WebKitTestRunner -
>+14:45:21.670 51856
>+14:45:21.670 51856 Collecting tests ...
>+14:45:21.692 51856 Parsing expectations ...
>+14:45:21.983 51856 Found 109 tests; running 109, skipping 0.
>+14:45:21.984 51856
>+14:45:21.984 51856 Running 109 tests
>+14:45:21.984 51856
>+14:45:21.984 51856 Checking build ...
>+14:45:21.984 51856 Starting helper ...
>+14:45:21.984 51856 Starting layout helper /Volumes/Data/Code/Customer/OpenSource/WebKitBuild/Debug/LayoutTestHelper
>+14:45:22.182 51856 Resetting persistent preferences
>+14:45:22.407 51856 "defaults delete DumpRenderTree" took 0.22s
>+14:45:22.602 51856 "defaults delete WebKitTestRunner" took 0.19s
>+14:45:22.602 51856 Checking system dependencies ...
>+14:45:22.625 51856 Expect:   109 passes   (109 now, 0 wontfix)
>+14:45:22.625 51856 Expect:     0 failures (  0 now, 0 wontfix)
>+14:45:22.625 51856 Expect:     0 flaky    (  0 now, 0 wontfix)
>+14:45:22.625 51856
>+14:45:22.625 51856 Sharding tests ...
>+14:45:22.625 51856 Running 6 WebKitTestRunners in parallel.
>+14:45:22.625 51856 (11 shards).
>+14:45:22.625 51856
>+14:45:22.625 51856 Starting 6 workers ...
>+14:45:22.675 51882 worker/0 starting
>+14:45:22.776 51885 worker/1 starting
>+14:45:22.880 51887 worker/2 starting
>+14:45:22.983 51889 worker/3 starting
>+14:45:23.086 51891 worker/4 starting
>+14:45:23.189 51893 worker/5 starting
>+14:45:25.496 51887 worker/2 css1/cascade/cascade_order.html output stderr lines:
>+14:45:25.496 51887   _RegisterApplication(), FAILED TO establish the default connection to the WindowServer, _CGSDefaultConnection() is NULL.
>+14:45:25.497 51887 worker/2 css1/cascade/cascade_order.html passed
>+14:45:25.499 51891 worker/4 css1/color_and_background/background.html output stderr lines:
>+14:45:25.499 51891   _RegisterApplication(), FAILED TO establish the default connection to the WindowServer, _CGSDefaultConnection() is NULL.
>+14:45:25.500 51891 worker/4 css1/color_and_background/background.html passed
>+14:45:25.506 51889 worker/3 css1/classification/display.html output stderr lines:
>+14:45:25.506 51889   _RegisterApplication(), FAILED TO establish the default connection to the WindowServer, _CGSDefaultConnection() is NULL.
>+14:45:25.507 51889 worker/3 css1/classification/display.html passed
>+14:45:25.606 51893 worker/5 css1/conformance/forward_compatible_parsing.html output stderr lines:
>+14:45:25.606 51893   _RegisterApplication(), FAILED TO establish the default connection to the WindowServer, _CGSDefaultConnection() is NULL.
>+14:45:25.608 51893 worker/5 css1/conformance/forward_compatible_parsing.html passed
>+14:45:25.681 51887 worker/2 css1/cascade/important.html passed
>+14:45:25.684 51882 worker/0 css1/basic/class_as_selector.html output stderr lines:
>+14:45:25.685 51882   _RegisterApplication(), FAILED TO establish the default connection to the WindowServer, _CGSDefaultConnection() is NULL.
>+14:45:25.686 51882 worker/0 css1/basic/class_as_selector.html passed
>+14:45:25.695 51889 worker/3 css1/classification/list_style.html passed
>+14:45:25.714 51891 worker/4 css1/color_and_background/background_attachment.html passed
>+14:45:25.824 51887 worker/2 css1/formatting_model/canvas.html passed
>+14:45:25.869 51889 worker/3 css1/classification/list_style_image.html passed
>+14:45:25.910 51885 worker/1 css1/box_properties/acid_test.html output stderr lines:
>+14:45:25.910 51885   _RegisterApplication(), FAILED TO establish the default connection to the WindowServer, _CGSDefaultConnection() is NULL.
>+14:45:25.912 51885 worker/1 css1/box_properties/acid_test.html passed
>+14:45:25.925 51891 worker/4 css1/color_and_background/background_color.html passed
>+14:45:26.019 51889 worker/3 css1/classification/list_style_position.html passed
>+14:45:26.070 51882 worker/0 css1/basic/comments.html passed
>+14:45:26.079 51893 worker/5 css1/font_properties/font.html passed
>+14:45:26.089 51891 worker/4 css1/color_and_background/background_image.html passed
>+14:45:26.107 51885 worker/1 killing driver
>+14:45:26.107 51885 worker/1 css1/box_properties/border.html failed:
>+09:30:46.509 51885 worker/1  test timed out
>+09:30:46.509 51885 worker/1  text diff
>+09:30:46.509 51885 worker/1 css1/box_properties/border.html output stderr lines:
>+14:45:25.910 51885   Something.
>+14:45:26.182 51889 worker/3 css1/classification/list_style_type.html passed
>+14:45:26.206 51887 worker/2 css1/formatting_model/floating_elements.html passed
>+14:45:26.234 51891 worker/4 css1/color_and_background/background_position.html passed
>+14:45:26.235 51882 worker/0 css1/basic/containment.html passed
>+14:45:26.242 51893 worker/5 css1/font_properties/font_family.html passed
>+14:45:26.250 51885 worker/1 css1/box_properties/border_bottom.html passed
>+14:45:26.357 51889 worker/3 css1/classification/white_space.html passed
>+14:45:26.382 51887 worker/2 css1/formatting_model/height_of_lines.html passed
>+14:45:26.394 51885 worker/1 css1/box_properties/border_bottom_inline.html passed
>+14:45:26.420 51882 worker/0 css1/basic/contextual_selectors.html passed
>+14:45:26.450 51893 worker/5 css1/font_properties/font_size.html passed
>+14:45:26.510 51891 worker/4 css1/color_and_background/background_repeat.html passed
>+14:45:26.524 51889 worker/3 css1/pseudo/anchor.html passed
>+14:45:26.528 51885 worker/1 css1/box_properties/border_bottom_width.html passed
>+14:45:26.528 51882 worker/0 css1/basic/grouping.html passed
>+14:45:26.546 51887 worker/2 css1/formatting_model/horizontal_formatting.html passed
>+14:45:26.587 51893 worker/5 css1/font_properties/font_style.html passed
>+14:45:26.642 51891 worker/4 css1/color_and_background/color.html passed
>+14:45:26.668 51882 worker/0 css1/basic/id_as_selector.html passed
>+14:45:26.725 51887 worker/2 css1/formatting_model/inline_elements.html passed
>+14:45:26.726 51885 worker/1 css1/box_properties/border_bottom_width_inline.html passed
>+14:45:26.765 51893 worker/5 css1/font_properties/font_variant.html passed
>+14:45:26.768 51891 worker/4 css1/text_properties/letter_spacing.html passed
>+14:45:26.848 51885 worker/1 css1/box_properties/border_color.html passed
>+14:45:26.851 51882 worker/0 css1/basic/inheritance.html passed
>+14:45:26.891 51887 worker/2 css1/formatting_model/replaced_elements.html passed
>+14:45:26.894 51889 worker/3 css1/pseudo/firstletter-surrogate.html passed
>+14:45:26.919 51893 worker/5 css1/font_properties/font_weight.html passed
>+14:45:26.919 51893 worker/5 exiting
>+14:45:26.919 51893 worker/5 cleaning up
>+14:45:26.919 51893 worker/5 killing driver
>+14:45:27.006 51885 worker/1 css1/box_properties/border_color_inline.html passed
>+14:45:27.014 51891 worker/4 css1/text_properties/line_height.html passed
>+14:45:27.049 51882 worker/0 css1/units/color_units.html passed
>+14:45:27.054 51889 worker/3 css1/pseudo/firstletter.html passed
>+14:45:27.054 51887 worker/2 css1/formatting_model/vertical_formatting.html passed
>+14:45:27.054 51887 worker/2 exiting
>+14:45:27.054 51887 worker/2 cleaning up
>+14:45:27.054 51887 worker/2 killing driver
>+14:45:27.130 51891 worker/4 css1/text_properties/text_align.html passed
>+14:45:27.145 51885 worker/1 css1/box_properties/border_inline.html passed
>+14:45:27.232 51882 worker/0 css1/units/length_units.html passed
>+14:45:27.266 51889 worker/3 css1/pseudo/firstline.html passed
>+14:45:27.269 51885 worker/1 css1/box_properties/border_left.html passed
>+14:45:27.297 51891 worker/4 css1/text_properties/text_decoration.html passed
>+14:45:27.320 51882 worker/0 css1/units/percentage_units.html passed
>+14:45:27.386 51885 worker/1 css1/box_properties/border_left_inline.html passed
>+14:45:27.403 51889 worker/3 css1/pseudo/multiple_pseudo_elements.html passed
>+14:45:27.430 51891 worker/4 css1/text_properties/text_indent.html passed
>+14:45:27.480 51882 worker/0 css1/units/rounding.html passed
>+14:45:27.507 51885 worker/1 css1/box_properties/border_left_width.html passed
>+14:45:27.553 51891 worker/4 css1/text_properties/text_transform.html passed
>+14:45:27.524 51889 worker/3 css1/pseudo/pseudo_elements_in_selectors.html passed
>+14:45:27.525 51889 worker/3 exiting
>+14:45:27.525 51889 worker/3 cleaning up
>+14:45:27.525 51889 worker/3 killing driver
>+14:45:27.630 51882 worker/0 css1/units/urls.html passed
>+14:45:27.654 51885 worker/1 css1/box_properties/border_left_width_inline.html passed
>+14:45:27.744 51885 worker/1 css1/box_properties/border_right.html passed
>+14:45:27.721 51882 worker/0 css1/units/zero-duration-without-units.html passed
>+14:45:27.722 51882 worker/0 exiting
>+14:45:27.722 51882 worker/0 cleaning up
>+14:45:27.722 51882 worker/0 killing driver
>+14:45:27.810 51891 worker/4 css1/text_properties/vertical_align.html passed
>+14:45:27.862 51885 worker/1 css1/box_properties/border_right_inline.html passed
>+14:45:27.895 51891 worker/4 css1/text_properties/word_spacing.html passed
>+14:45:27.396 51885 worker/4 css1/box_properties/border_top_width.html passed
>+14:45:27.896 51891 worker/4 exiting
>+14:45:27.896 51891 worker/4 cleaning up
>+14:45:27.896 51891 worker/4 killing driver
>+14:45:27.970 51885 worker/1 css1/box_properties/border_right_width.html passed
>+14:45:28.060 51885 worker/1 css1/box_properties/border_right_width_inline.html passed
>+14:45:28.133 51885 worker/1 css1/box_properties/border_style.html passed
>+14:45:28.197 51885 worker/1 css1/box_properties/border_style_inline.html passed
>+14:45:28.271 51885 worker/1 css1/box_properties/border_top.html passed
>+14:45:28.334 51885 worker/1 css1/box_properties/border_top_inline.html passed
>+14:45:28.463 51885 worker/1 css1/box_properties/border_top_width_inline.html passed
>+14:45:28.540 51885 worker/1 css1/box_properties/border_width.html passed
>+14:45:28.610 51885 worker/1 css1/box_properties/border_width_inline.html passed
>+14:45:28.684 51885 worker/1 css1/box_properties/clear.html passed
>+14:45:28.766 51885 worker/1 css1/box_properties/clear_float.html passed
>+14:45:28.830 51885 worker/1 css1/box_properties/float.html passed
>+14:45:28.936 51885 worker/1 css1/box_properties/float_elements_in_series.html passed
>+14:45:29.028 51885 worker/1 css1/box_properties/float_margin.html passed
>+14:45:29.135 51885 worker/1 css1/box_properties/float_on_text_elements.html passed
>+14:45:29.212 51885 worker/1 css1/box_properties/height.html passed
>+14:45:29.291 51885 worker/1 css1/box_properties/margin.html passed
>+14:45:29.366 51885 worker/1 css1/box_properties/margin_bottom.html passed
>+14:45:29.434 51885 worker/1 css1/box_properties/margin_bottom_inline.html passed
>+14:45:29.507 51885 worker/1 css1/box_properties/margin_inline.html passed
>+14:45:29.591 51885 worker/1 css1/box_properties/margin_left.html passed
>+14:45:29.666 51885 worker/1 css1/box_properties/margin_left_inline.html passed
>+14:45:29.752 51885 worker/1 css1/box_properties/margin_right.html passed
>+14:45:29.832 51885 worker/1 css1/box_properties/margin_right_inline.html passed
>+14:45:29.923 51885 worker/1 css1/box_properties/margin_top.html passed
>+14:45:29.995 51885 worker/1 css1/box_properties/margin_top_inline.html passed
>+14:45:30.061 51885 worker/1 css1/box_properties/padding.html passed
>+14:45:30.131 51885 worker/1 css1/box_properties/padding_bottom.html passed
>+14:45:30.204 51885 worker/1 css1/box_properties/padding_bottom_inline.html passed
>+14:45:30.277 51885 worker/1 css1/box_properties/padding_inline.html passed
>+14:45:30.351 51885 worker/1 css1/box_properties/padding_left.html passed
>+14:45:30.420 51885 worker/1 css1/box_properties/padding_left_inline.html passed
>+14:45:30.500 51885 worker/1 css1/box_properties/padding_right.html passed
>+14:45:30.574 51885 worker/1 css1/box_properties/padding_right_inline.html passed
>+14:45:30.663 51885 worker/1 css1/box_properties/padding_top.html passed
>+14:45:30.743 51885 worker/1 css1/box_properties/padding_top_inline.html passed
>+14:45:30.820 51856
>+14:45:30.820 51885 worker/1 css1/box_properties/width.html passed
>+14:45:30.820 51885 worker/1 exiting
>+14:45:30.820 51885 worker/1 cleaning up
>+14:45:30.820 51885 worker/1 killing driver
>+14:45:30.857 51856 worker/1 cleaning up
>+14:45:30.858 51856 worker/1 cleaning up
>+14:45:30.862 51856 Flushing stdout
>+14:45:30.862 51856 Flushing stderr
>+14:45:30.862 51856 Stopping helper
>+14:45:30.863 51856 Stopping LayoutTestHelper
>+14:45:31.004 51856 Cleaning up port
>+14:45:31.004 51856 looking for new crash logs
>+14:45:31.006 51856 summarizing results
>+14:45:31.039 51856 "ruby --version" took 0.03s
>+14:45:31.041 51856 Test timing:
>+14:45:31.041 51856     9.02 total testing time
>+14:45:31.041 51856
>+14:45:31.041 51856 Thread timing:
>+14:45:31.041 51856       worker/3:    12 tests,   4.51 secs
>+14:45:31.041 51856       worker/2:     9 tests,   4.15 secs
>+14:45:31.041 51856       worker/1:    53 tests,   7.93 secs
>+14:45:31.041 51856       worker/0:    13 tests,   5.01 secs
>+14:45:31.041 51856       worker/5:     7 tests,   3.71 secs
>+14:45:31.041 51856       worker/4:    15 tests,   4.77 secs
>+14:45:31.041 51856     30.08 cumulative,   5.01 optimal
>+14:45:31.041 51856
>+14:45:31.041 51856 PER TEST TIME IN TESTSHELL (seconds):
>+14:45:31.041 51856   Median:           0.128
>+14:45:31.041 51856   Mean:             0.272
>+14:45:31.042 51856   90th percentile:  0.272
>+14:45:31.042 51856   99th percentile:  2.978
>+14:45:31.042 51856   Standard dev:     0.272
>+14:45:31.042 51856
>+14:45:31.042 51856
>+14:45:31.042 51856 10 slowest tests that are not marked as SLOW and did not timeout/crash:
>+14:45:31.042 51856   css1/box_properties/acid_test.html took 3.1 seconds
>+14:45:31.042 51856   css1/basic/class_as_selector.html took 3.0 seconds
>+14:45:31.042 51856   css1/cascade/cascade_order.html took 2.6 seconds
>+14:45:31.042 51856   css1/classification/display.html took 2.5 seconds
>+14:45:31.042 51856   css1/color_and_background/background.html took 2.4 seconds
>+14:45:31.042 51856   css1/conformance/forward_compatible_parsing.html took 2.4 seconds
>+14:45:31.042 51856   css1/font_properties/font.html took 0.5 seconds
>+14:45:31.042 51856   css1/basic/comments.html took 0.4 seconds
>+14:45:31.042 51856   css1/formatting_model/floating_elements.html took 0.4 seconds
>+14:45:31.042 51856   css1/pseudo/firstletter-surrogate.html took 0.4 seconds
>+14:45:31.042 51856
>+14:45:31.042 51856 Tests marked as SLOW:
>+14:45:31.042 51856
>+14:45:31.042 51856 Tests that timed out or crashed:
>+14:45:31.042 51856
>+14:45:31.042 51856 Time to process slowest subdirectories:
>+14:45:31.042 51856   css1/basic took 4.2 seconds to run 7 tests.
>+14:45:31.043 51856   css1/box_properties took 7.9 seconds to run 53 tests.
>+14:45:31.043 51856   css1/cascade took 2.8 seconds to run 2 tests.
>+14:45:31.043 51856   css1/classification took 3.4 seconds to run 6 tests.
>+14:45:31.043 51856   css1/color_and_background took 3.5 seconds to run 7 tests.
>+14:45:31.043 51856   css1/conformance took 2.4 seconds to run 1 tests.
>+14:45:31.043 51856   css1/font_properties took 1.3 seconds to run 6 tests.
>+14:45:31.043 51856   css1/formatting_model took 1.4 seconds to run 7 tests.
>+14:45:31.043 51856   css1/pseudo took 1.2 seconds to run 6 tests.
>+14:45:31.043 51856   css1/text_properties took 1.2 seconds to run 8 tests.
>+14:45:31.043 51856   css1/units took 0.9 seconds to run 6 tests.
>+14:45:31.043 51856
>+14:45:31.043 51856
>+14:45:31.043 51856 All 109 tests ran as expected.
>+14:45:31.043 51856
>+14:45:31.043 51856 Writing JSON files in /Volumes/Data/Code/Customer/OpenSource/WebKitBuild/Debug/layout-test-results.
>+14:45:31.049 51856 Finished writing JSON file for the test results server.
>+14:45:31.051 51856 Testing completed, Exit status: 0
>+=> Results: 109/109 tests passed (100.0%)
>+
>+=> Tests to be fixed (0):
>+
>+=> Tests that will only be fixed if they crash (WONTFIX) (0):
>+"""
>+    FAILED_STDOUT = """14:45:21.473 51856 "sysctl -n hw.memsize" took 0.01s
>+14:45:21.667 51856 "perl Tools/Scripts/webkit-build-directory --configuration --debug --mac" took 0.19s
>+14:45:21.669 51856 Using port 'mac-mojave-wk2'
>+14:45:21.669 51856 Test configuration: <mojave, x86_64, debug>
>+14:45:21.669 51856 Placing test results in /Volumes/Data/Code/Customer/OpenSource/WebKitBuild/Debug/layout-test-results
>+14:45:21.669 51856 Baseline search path: platform/mac-mojave-wk2 -> platform/mac-mojave -> platform/mac-wk2 -> platform/mac -> platform/wk2 -> generic
>+14:45:21.669 51856 Using Debug build
>+14:45:21.669 51856 Pixel tests disabled
>+14:45:21.669 51856 Regular timeout: 30000, slow test timeout: 150000
>+14:45:21.670 51856 Command line: /Volumes/Data/Code/Customer/OpenSource/WebKitBuild/Debug/WebKitTestRunner -
>+14:45:21.670 51856
>+14:45:21.670 51856 Collecting tests ...
>+14:45:21.692 51856 Parsing expectations ...
>+14:45:21.983 51856 Found 109 tests; running 109, skipping 0.
>+14:45:21.984 51856
>+14:45:21.984 51856 Running 109 tests
>+14:45:21.984 51856
>+14:45:21.984 51856 Checking build ...
>+14:45:21.984 51856 Starting helper ...
>+14:45:21.984 51856 Starting layout helper /Volumes/Data/Code/Customer/OpenSource/WebKitBuild/Debug/LayoutTestHelper
>+14:45:22.182 51856 Resetting persistent preferences
>+14:45:22.407 51856 "defaults delete DumpRenderTree" took 0.22s
>+14:45:22.602 51856 "defaults delete WebKitTestRunner" took 0.19s
>+14:45:22.602 51856 Checking system dependencies ...
>+14:45:22.625 51856 Expect:   109 passes   (109 now, 0 wontfix)
>+14:45:22.625 51856 Expect:     0 failures (  0 now, 0 wontfix)
>+14:45:22.625 51856 Expect:     0 flaky    (  0 now, 0 wontfix)
>+14:45:22.625 51856
>+14:45:22.625 51856 Sharding tests ...
>+14:45:22.625 51856 Running 6 WebKitTestRunners in parallel.
>+14:45:22.625 51856 (11 shards).
>+14:45:22.625 51856
>+14:45:22.625 51856 Starting 6 workers ...
>+14:45:22.675 51882 worker/0 starting
>+14:45:22.776 51885 worker/1 starting
>+14:45:22.880 51887 worker/2 starting
>+14:45:22.983 51889 worker/3 starting
>+14:45:23.086 51891 worker/4 starting
>+14:45:23.189 51893 worker/5 starting
>+14:45:25.496 51887 worker/2 css1/cascade/cascade_order.html output stderr lines:
>+14:45:25.496 51887   _RegisterApplication(), FAILED TO establish the default connection to the WindowServer, _CGSDefaultConnection() is NULL.
>+14:45:25.497 51887 worker/2 css1/cascade/cascade_order.html passed
>+14:45:25.499 51891 worker/4 css1/color_and_background/background.html output stderr lines:
>+14:45:25.499 51891   _RegisterApplication(), FAILED TO establish the default connection to the WindowServer, _CGSDefaultConnection() is NULL.
>+14:45:25.500 51891 worker/4 css1/color_and_background/background.html passed
>+14:45:25.506 51889 worker/3 css1/classification/display.html output stderr lines:
>+14:45:25.506 51889   _RegisterApplication(), FAILED TO establish the default connection to the WindowServer, _CGSDefaultConnection() is NULL.
>+14:45:25.507 51889 worker/3 css1/classification/display.html passed
>+14:45:25.606 51893 worker/5 css1/conformance/forward_compatible_parsing.html output stderr lines:
>+14:45:25.606 51893   _RegisterApplication(), FAILED TO establish the default connection to the WindowServer, _CGSDefaultConnection() is NULL.
>+14:45:25.608 51893 worker/5 css1/conformance/forward_compatible_parsing.html passed
>+14:45:25.681 51887 worker/2 css1/cascade/important.html passed
>+14:45:25.684 51882 worker/0 css1/basic/class_as_selector.html output stderr lines:
>+14:45:25.685 51882   _RegisterApplication(), FAILED TO establish the default connection to the WindowServer, _CGSDefaultConnection() is NULL.
>+14:45:25.686 51882 worker/0 css1/basic/class_as_selector.html passed
>+14:45:25.695 51889 worker/3 css1/classification/list_style.html passed
>+14:45:25.714 51891 worker/4 css1/color_and_background/background_attachment.html passed
>+14:45:25.824 51887 worker/2 css1/formatting_model/canvas.html passed
>+14:45:25.869 51889 worker/3 css1/classification/list_style_image.html passed
>+14:45:25.910 51885 worker/1 css1/box_properties/acid_test.html output stderr lines:
>+14:45:25.910 51885   _RegisterApplication(), FAILED TO establish the default connection to the WindowServer, _CGSDefaultConnection() is NULL.
>+14:45:25.912 51885 worker/1 css1/box_properties/acid_test.html passed
>+14:45:25.925 51891 worker/4 css1/color_and_background/background_color.html passed
>+14:45:26.019 51889 worker/3 css1/classification/list_style_position.html passed
>+14:45:26.070 51882 worker/0 css1/basic/comments.html passed
>+14:45:26.079 51893 worker/5 css1/font_properties/font.html passed
>+14:45:26.089 51891 worker/4 css1/color_and_background/background_image.html passed
>+14:45:26.107 51885 worker/1 killing driver
>+14:45:26.107 51885 worker/1 css1/box_properties/border.html failed:
>+09:30:46.509 51885 worker/1  test timed out
>+09:30:46.509 51885 worker/1  text diff
>+09:30:46.509 51885 worker/1 css1/box_properties/border.html output stderr lines:
>+14:45:25.910 51885   Something.
>+14:45:26.182 51889 worker/3 css1/classification/list_style_type.html passed
>+14:45:26.206 51887 worker/2 css1/formatting_model/floating_elements.html passed
>+14:45:26.234 51891 worker/4 css1/color_and_background/background_position.html passed
>+14:45:26.235 51882 worker/0 css1/basic/containment.html passed
>+14:45:26.242 51893 worker/5 css1/font_properties/font_family.html passed
>+14:45:26.250 51885 worker/1 css1/box_properties/border_bottom.html passed
>+14:45:26.357 51889 worker/3 css1/classification/white_space.html passed
>+14:45:26.382 51887 worker/2 css1/formatting_model/height_of_lines.html passed
>+14:45:26.394 51885 worker/1 css1/box_properties/border_bottom_inline.html passed
>+14:45:26.420 51882 worker/0 css1/basic/contextual_selectors.html passed
>+14:45:26.450 51893 worker/5 css1/font_properties/font_size.html passed
>+14:45:26.510 51891 worker/4 css1/color_and_background/background_repeat.html passed
>+14:45:26.524 51889 worker/3 css1/pseudo/anchor.html passed
>+14:45:26.528 51885 worker/1 css1/box_properties/border_bottom_width.html passed
>+14:45:26.528 51882 worker/0 css1/basic/grouping.html passed
>+14:45:26.546 51887 worker/2 css1/formatting_model/horizontal_formatting.html passed
>+14:45:26.587 51893 worker/5 css1/font_properties/font_style.html passed
>+14:45:26.642 51891 worker/4 css1/color_and_background/color.html passed
>+14:45:26.668 51882 worker/0 css1/basic/id_as_selector.html passed
>+14:45:26.725 51887 worker/2 css1/formatting_model/inline_elements.html passed
>+14:45:26.726 51885 worker/1 css1/box_properties/border_bottom_width_inline.html passed
>+14:45:26.765 51893 worker/5 css1/font_properties/font_variant.html passed
>+14:45:26.768 51891 worker/4 css1/text_properties/letter_spacing.html passed
>+14:45:26.848 51885 worker/1 css1/box_properties/border_color.html passed
>+14:45:26.851 51882 worker/0 css1/basic/inheritance.html passed
>+14:45:26.891 51887 worker/2 css1/formatting_model/replaced_elements.html passed
>+14:45:26.894 51889 worker/3 css1/pseudo/firstletter-surrogate.html passed
>+14:45:26.919 51893 worker/5 css1/font_properties/font_weight.html passed
>+14:45:26.919 51893 worker/5 exiting
>+14:45:26.919 51893 worker/5 cleaning up
>+14:45:26.919 51893 worker/5 killing driver
>+14:45:27.006 51885 worker/1 css1/box_properties/border_color_inline.html passed
>+14:45:27.014 51891 worker/4 css1/text_properties/line_height.html passed
>+14:45:27.049 51882 worker/0 css1/units/color_units.html passed
>+14:45:27.054 51889 worker/3 css1/pseudo/firstletter.html passed
>+14:45:27.054 51887 worker/2 css1/formatting_model/vertical_formatting.html passed
>+14:45:27.054 51887 worker/2 exiting
>+14:45:27.054 51887 worker/2 cleaning up
>+14:45:27.054 51887 worker/2 killing driver
>+14:45:27.130 51891 worker/1 css1/text_properties/text_align.html passed
>+14:45:27.145 51885 worker/1 css1/box_properties/border_inline.html passed
>+14:45:27.232 51882 worker/0 css1/units/length_units.html passed
>+14:45:27.266 51889 worker/3 css1/pseudo/firstline.html passed
>+14:45:27.269 51885 worker/1 css1/box_properties/border_left.html passed
>+14:45:27.297 51891 worker/4 css1/text_properties/text_decoration.html passed
>+14:45:27.320 51882 worker/0 css1/units/percentage_units.html passed
>+14:45:27.386 51885 worker/1 css1/box_properties/border_left_inline.html passed
>+14:45:27.403 51889 worker/3 css1/pseudo/multiple_pseudo_elements.html passed
>+14:45:27.430 51891 worker/4 css1/text_properties/text_indent.html passed
>+14:45:27.480 51882 worker/0 css1/units/rounding.html passed
>+14:45:27.507 51885 worker/1 css1/box_properties/border_left_width.html passed
>+14:45:27.553 51891 worker/4 css1/text_properties/text_transform.html passed
>+14:45:27.524 51889 worker/3 css1/pseudo/pseudo_elements_in_selectors.html passed
>+14:45:27.525 51889 worker/3 exiting
>+14:45:27.525 51889 worker/3 cleaning up
>+14:45:27.525 51889 worker/3 killing driver
>+14:45:27.630 51882 worker/0 css1/units/urls.html passed
>+14:45:27.654 51885 worker/1 css1/box_properties/border_left_width_inline.html passed
>+14:45:27.744 51885 worker/1 css1/box_properties/border_right.html passed
>+14:45:27.721 51882 worker/0 css1/units/zero-duration-without-units.html passed
>+14:45:27.722 51882 worker/0 exiting
>+14:45:27.722 51882 worker/0 cleaning up
>+14:45:27.722 51882 worker/0 killing driver
>+14:45:27.810 51891 worker/4 css1/text_properties/vertical_align.html passed
>+14:45:27.862 51885 worker/1 css1/box_properties/border_right_inline.html passed
>+14:45:27.896 51891 worker/4 css1/text_properties/word_spacing.html passed
>+14:45:27.896 51891 worker/4 exiting
>+14:45:27.896 51891 worker/4 cleaning up
>+14:45:27.896 51891 worker/4 killing driver
>+14:45:27.970 51885 worker/1 css1/box_properties/border_right_width.html passed
>+14:45:28.060 51885 worker/1 css1/box_properties/border_right_width_inline.html passed
>+14:45:28.133 51885 worker/1 css1/box_properties/border_style.html passed
>+14:45:28.197 51885 worker/1 css1/box_properties/border_style_inline.html passed
>+14:45:28.271 51885 worker/1 css1/box_properties/border_top.html passed
>+14:45:28.334 51885 worker/1 css1/box_properties/border_top_inline.html passed
>+14:45:28.398 51885 worker/1 css1/box_properties/border_top_width.html passed
>+14:45:28.463 51885 worker/1 css1/box_properties/border_top_width_inline.html passed
>+14:45:28.540 51885 worker/1 css1/box_properties/border_width.html passed
>+14:45:28.610 51885  [81/109] css1/box_properties/border_width_inline.html failed unexpectedly (text diff)
>+14:45:28.611 51885 worker/1 css1/box_properties/border_width_inline.html failed:
>+14:45:28.612 51885 worker/1  text diff
>+14:45:28.684 51885 worker/1 css1/box_properties/clear.html passed
>+14:45:28.766 51885 worker/1 css1/box_properties/clear_float.html passed
>+14:45:28.830 51885 worker/1 css1/box_properties/float.html passed
>+14:45:28.936 51885 worker/1 css1/box_properties/float_elements_in_series.html passed
>+14:45:29.028 51885 worker/1 css1/box_properties/float_margin.html passed
>+14:45:29.135 51885 worker/1 css1/box_properties/float_on_text_elements.html passed
>+14:45:29.212 51885 worker/1 css1/box_properties/height.html passed
>+14:45:29.291 51885 worker/1 css1/box_properties/margin.html passed
>+14:45:29.366 51885 worker/1 css1/box_properties/margin_bottom.html passed
>+14:45:29.434 51885 worker/1 css1/box_properties/margin_bottom_inline.html passed
>+14:45:29.507 51885 worker/1 css1/box_properties/margin_inline.html passed
>+14:45:29.591 51885 worker/1 css1/box_properties/margin_left.html passed
>+14:45:29.666 51885 worker/1 css1/box_properties/margin_left_inline.html passed
>+14:45:29.752 51885 worker/1 css1/box_properties/margin_right.html passed
>+14:45:29.832 51885 worker/1 css1/box_properties/margin_right_inline.html passed
>+14:45:29.923 51885 worker/1 css1/box_properties/margin_top.html passed
>+14:45:29.995 51885 worker/1 css1/box_properties/margin_top_inline.html passed
>+14:45:30.061 51885 worker/1 css1/box_properties/padding.html passed
>+14:45:30.131 51885 worker/1 css1/box_properties/padding_bottom.html passed
>+14:45:30.204 51885 worker/1 css1/box_properties/padding_bottom_inline.html passed
>+14:45:30.277 51885 worker/1 css1/box_properties/padding_inline.html passed
>+14:45:30.351 51885 worker/1 css1/box_properties/padding_left.html passed
>+14:45:30.420 51885 worker/1 css1/box_properties/padding_left_inline.html passed
>+14:45:30.500 51885 worker/1 css1/box_properties/padding_right.html passed
>+14:45:30.574 51885 worker/1 css1/box_properties/padding_right_inline.html passed
>+14:45:30.663 51885 worker/1 css1/box_properties/padding_top.html passed
>+14:45:30.743 51885 worker/1 css1/box_properties/padding_top_inline.html passed
>+14:45:30.820 51856
>+14:45:30.820 51885 worker/1 css1/box_properties/width.html passed
>+14:45:30.820 51885 worker/1 exiting
>+14:45:30.820 51885 worker/1 cleaning up
>+14:45:30.820 51885 worker/1 killing driver
>+14:45:30.857 51856 worker/1 cleaning up
>+14:45:30.858 51856 worker/1 cleaning up
>+14:45:30.862 51856 Flushing stdout
>+14:45:30.862 51856 Flushing stderr
>+14:45:30.862 51856 Stopping helper
>+14:45:30.863 51856 Stopping LayoutTestHelper
>+14:45:31.004 51856 Cleaning up port
>+14:45:31.004 51856 looking for new crash logs
>+14:45:31.006 51856 summarizing results
>+14:45:31.039 51856 "ruby --version" took 0.03s
>+14:45:31.041 51856 Test timing:
>+14:45:31.041 51856     9.02 total testing time
>+14:45:31.041 51856
>+14:45:31.041 51856 Thread timing:
>+14:45:31.041 51856       worker/3:    12 tests,   4.51 secs
>+14:45:31.041 51856       worker/2:     9 tests,   4.15 secs
>+14:45:31.041 51856       worker/1:    53 tests,   7.93 secs
>+14:45:31.041 51856       worker/0:    13 tests,   5.01 secs
>+14:45:31.041 51856       worker/5:     7 tests,   3.71 secs
>+14:45:31.041 51856       worker/4:    15 tests,   4.77 secs
>+14:45:31.041 51856     30.08 cumulative,   5.01 optimal
>+14:45:31.041 51856
>+14:45:31.041 51856 PER TEST TIME IN TESTSHELL (seconds):
>+14:45:31.041 51856   Median:           0.128
>+14:45:31.041 51856   Mean:             0.272
>+14:45:31.042 51856   90th percentile:  0.272
>+14:45:31.042 51856   99th percentile:  2.978
>+14:45:31.042 51856   Standard dev:     0.272
>+14:45:31.042 51856
>+14:45:31.042 51856
>+14:45:31.042 51856 10 slowest tests that are not marked as SLOW and did not timeout/crash:
>+14:45:31.042 51856   css1/box_properties/acid_test.html took 3.1 seconds
>+14:45:31.042 51856   css1/basic/class_as_selector.html took 3.0 seconds
>+14:45:31.042 51856   css1/cascade/cascade_order.html took 2.6 seconds
>+14:45:31.042 51856   css1/classification/display.html took 2.5 seconds
>+14:45:31.042 51856   css1/color_and_background/background.html took 2.4 seconds
>+14:45:31.042 51856   css1/conformance/forward_compatible_parsing.html took 2.4 seconds
>+14:45:31.042 51856   css1/font_properties/font.html took 0.5 seconds
>+14:45:31.042 51856   css1/basic/comments.html took 0.4 seconds
>+14:45:31.042 51856   css1/formatting_model/floating_elements.html took 0.4 seconds
>+14:45:31.042 51856   css1/pseudo/firstletter-surrogate.html took 0.4 seconds
>+14:45:31.042 51856
>+14:45:31.042 51856 Tests marked as SLOW:
>+14:45:31.042 51856
>+14:45:31.042 51856 Tests that timed out or crashed:
>+14:45:31.042 51856
>+14:45:31.042 51856 Time to process slowest subdirectories:
>+14:45:31.042 51856   css1/basic took 4.2 seconds to run 7 tests.
>+14:45:31.043 51856   css1/box_properties took 7.9 seconds to run 53 tests.
>+14:45:31.043 51856   css1/cascade took 2.8 seconds to run 2 tests.
>+14:45:31.043 51856   css1/classification took 3.4 seconds to run 6 tests.
>+14:45:31.043 51856   css1/color_and_background took 3.5 seconds to run 7 tests.
>+14:45:31.043 51856   css1/conformance took 2.4 seconds to run 1 tests.
>+14:45:31.043 51856   css1/font_properties took 1.3 seconds to run 6 tests.
>+14:45:31.043 51856   css1/formatting_model took 1.4 seconds to run 7 tests.
>+14:45:31.043 51856   css1/pseudo took 1.2 seconds to run 6 tests.
>+14:45:31.043 51856   css1/text_properties took 1.2 seconds to run 8 tests.
>+14:45:31.043 51856   css1/units took 0.9 seconds to run 6 tests.
>+14:45:31.043 51856
>+14:45:31.043 51856
>+14:45:31.043 51856 108 tests ran as expected, 1 didn't:
>+14:45:31.043 51856
>+14:45:31.043 51856 Writing JSON files in /Volumes/Data/Code/Customer/OpenSource/WebKitBuild/Debug/layout-test-results.
>+14:45:31.049 51856 Finished writing JSON file for the test results server.
>+14:45:31.051 51856 Testing completed, Exit status: 1
>+=> Results: 108/109 tests passed (99.1%)
>+
>+=> Tests to be fixed (1):
>+
>+=> Tests that will only be fixed if they crash (WONTFIX) (0):
>+
>+
>+Regressions: Unexpected text-only failures (1)
>+  css1/box_properties/border_width_inline.html [ Failure ]
>+
>+------------------------------------------------------------------------------
>+"""
>+
>+    def create_mock_host(self):
>+        host = MockHost()
>+        host.filesystem.write_text_file('success.txt', self.SUCCESS_STDOUT)
>+        host.filesystem.write_text_file('failed.txt', self.FAILED_STDOUT)
>+        return host
>+
>+    def test_single_failure(self):
>+        OutputCapture().assert_outputs(
>+            self, lambda: analyze_stdouts(['failed.txt'], focus='css1/box_properties/border_width_inline.html', host=self.create_mock_host()), [],
>+            expected_stdout="""css1/box_properties/border_right_width_inline.html
>+css1/box_properties/border_style.html
>+css1/box_properties/border_style_inline.html
>+css1/box_properties/border_top.html
>+css1/box_properties/border_top_inline.html
>+css1/box_properties/border_top_width.html
>+css1/box_properties/border_top_width_inline.html
>+css1/box_properties/border_width.html
>+css1/box_properties/border_width_inline.html
>+""")
>+
>+    def test_single_failure_auto_focus(self):
>+        OutputCapture().assert_outputs(
>+            self, lambda: analyze_stdouts(['failed.txt'], host=self.create_mock_host()), [],
>+            expected_stdout="""css1/box_properties/border_right_width_inline.html
>+css1/box_properties/border_style.html
>+css1/box_properties/border_style_inline.html
>+css1/box_properties/border_top.html
>+css1/box_properties/border_top_inline.html
>+css1/box_properties/border_top_width.html
>+css1/box_properties/border_top_width_inline.html
>+css1/box_properties/border_width.html
>+css1/box_properties/border_width_inline.html
>+""")
>+
>+    def test_failure_and_success(self):
>+        OutputCapture().assert_outputs(
>+            self, lambda: analyze_stdouts(['failed.txt', 'success.txt'], focus='css1/box_properties/border_width_inline.html', host=self.create_mock_host()), [],
>+            expected_stdout='css1/box_properties/border_top_width.html\ncss1/box_properties/border_width_inline.html')
>+
>+    def test_failure_and_success_auto_focus(self):
>+        OutputCapture().assert_outputs(
>+            self, lambda: analyze_stdouts(['failed.txt', 'success.txt'], host=self.create_mock_host()), [],
>+            expected_stdout='css1/box_properties/border_top_width.html\ncss1/box_properties/border_width_inline.html')
>Index: Tools/Scripts/webkitpy/flakiness_analyzer/test_run.py
>===================================================================
>--- Tools/Scripts/webkitpy/flakiness_analyzer/test_run.py	(nonexistent)
>+++ Tools/Scripts/webkitpy/flakiness_analyzer/test_run.py	(working copy)
>@@ -0,0 +1,167 @@
>+# Copyright (C) 2018 Apple Inc. All rights reserved.
>+#
>+# Redistribution and use in source and binary forms, with or without
>+# modification, are permitted provided that the following conditions
>+# are met:
>+# 1.  Redistributions of source code must retain the above copyright
>+#    notice, this list of conditions and the following disclaimer.
>+# 2.  Redistributions in binary form must reproduce the above copyright
>+#    notice, this list of conditions and the following disclaimer in the
>+#    documentation and/or other materials provided with the distribution.
>+#
>+# THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS'' AND ANY
>+# EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
>+# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
>+# DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS BE LIABLE FOR ANY
>+# DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
>+# (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
>+# LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
>+# ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
>+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
>+# SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
>+
>+import re
>+
>+
>+class Test:
>+
>+    def __init__(self, name, result='passed', worker=None, did_kill=None):
>+        self.name = name
>+        self.result = result
>+        self.worker = worker
>+        self.expected = True
>+        self.did_stop = did_kill if did_kill != None else self.result in ['timed out', 'crashed']
>+
>+    def __str__(self):
>+        return self.name + ' [' + self.result + (']' if self.expected else ' (unexpected) ]')
>+
>+
>+class TestRun:
>+    PARSING_ERROR = 'Provided stdout does not look like the verbose output from run-webkit-tests'
>+    NUMBER_OF_WORKERS_REGEX = re.compile('Starting\s+(?P<number>\d+)\s+workers')
>+    TEST_RESULT_REGEX = re.compile('worker/(?P<worker>\d+)\s+(?P<test>[^\s]+)\s+(?P<result>.+)')
>+    TEST_RESULT_TYPE_REGEX = re.compile('worker/(?P<worker>\d+)\s+(?P<result>.+)')
>+    TEST_FINISHED_REGEX = re.compile('Testing completed, Exit status: \d+')
>+    UNEXPECTED_RESULT = re.compile('  (?P<test>[^\s]+)\s+\[\s+(?P<result>.+)\s+\]')
>+
>+    @staticmethod
>+    def _find_number_of_workers(stdout):
>+        found = None
>+        while not found:
>+            line = stdout.readline()
>+            if not line:
>+                raise RuntimeError(TestRun.PARSING_ERROR)
>+            found = TestRun.NUMBER_OF_WORKERS_REGEX.search(line)
>+        return int(found.group('number'))
>+
>+    @staticmethod
>+    def _find_next_test(stdout):
>+        found = None
>+        did_kill = None
>+        while not found:
>+            line = stdout.readline()
>+            if not line or TestRun.TEST_FINISHED_REGEX.search(line):
>+                return None
>+            found = TestRun.TEST_RESULT_REGEX.search(line)
>+            if found and ('output' in found.group('result') or 'stderr' in found.group('result')):
>+                found = None
>+                continue
>+            # This means that the next line should be the test that caused the failure
>+            if found and 'killing' == found.group('test'):
>+                did_kill = True
>+                found = None
>+                continue
>+            if found and 'cleaning' == found.group('test'):
>+                found = None
>+                continue
>+        result_type = found.group('result')
>+        if result_type == 'failed:':
>+            result_type = 'failed'
>+            while True:
>+                line = stdout.readline()
>+                if not line:
>+                    raise RuntimeError(TestRun.PARSING_ERROR)
>+                rslt = TestRun.TEST_RESULT_TYPE_REGEX.search(line)
>+                if not rslt:
>+                    break
>+                elif result_type == 'failed':
>+                    result_type = rslt.group('result')
>+            if 'crashed' in result_type:
>+                result_type = 'crashed'
>+                did_kill = True
>+        if result_type == 'test timed out':
>+            result_type == 'timed out'
>+
>+        return Test(name=found.group('test'), result=result_type, worker=int(found.group('worker')), did_kill=did_kill)
>+
>+    @staticmethod
>+    def from_stdout(stdout):
>+        result = TestRun()
>+
>+        number_of_workers = TestRun._find_number_of_workers(stdout)
>+        tests_for_worker = []
>+        for _ in xrange(number_of_workers):
>+            tests_for_worker.append([])
>+
>+        while True:
>+            test = TestRun._find_next_test(stdout)
>+            if not test:
>+                break
>+            tests_for_worker[test.worker].append(test)
>+
>+        for tests_on_worker in tests_for_worker:
>+            result.processes += TestRun.Process.from_test_list(tests_on_worker)
>+
>+        line = stdout.readline()
>+        while line:
>+            rg = TestRun.UNEXPECTED_RESULT.search(line)
>+            if rg:
>+                test_name = rg.group('test')
>+                for process in result.processes:
>+                    for test in process.tests:
>+                        if test.name == test_name:
>+                            test.expected = False
>+            line = stdout.readline()
>+        return result
>+
>+    class Process:
>+
>+        @staticmethod
>+        def from_test_list(tests):
>+            result = [TestRun.Process()]
>+            for test in tests:
>+                result[-1].tests.append(test)
>+                if test.did_stop:
>+                    result.append(TestRun.Process())
>+            return result
>+
>+        def __init__(self):
>+            self.tests = []
>+
>+        def contains_test(self, focus_list=[]):
>+            for test in self.tests:
>+                for focus in focus_list:
>+                    if test.name.startswith(focus):
>+                        return True
>+                if any(test.name.startswith(focus) for focus in focus_list):
>+                    return True
>+            return False
>+
>+        def failed_tests(self, focus_list=[]):
>+            result = []
>+            for test in self.tests:
>+                if focus_list and not any(test.name.startswith(focus) for focus in focus_list):
>+                    continue
>+                if not test.expected and test.result != 'passed':
>+                    result.append(test)
>+            return result
>+
>+    def __init__(self):
>+        self.processes = []
>+
>+    def processes_with_tests(self, focus_list=[]):
>+        result = []
>+        for process in self.processes:
>+            if process.contains_test(focus_list=focus_list):
>+                result.append(process)
>+        return result

Actions: View | Formatted Diff | Diff

Attachments on bug 188413: 346782 | 346807 | 346963