Attachment 418538 Details for Bug 220863 – Patch

[patch] Patch

bug-220863-20210127152703.patch (text/plain), 3.50 KB, created by Alicia Boya García on 2021-01-27 06:27:04 PST

(hide)

Description:

Filename:

MIME Type:

Creator: Alicia Boya García

Created: 2021-01-27 06:27:04 PST

Size: 3.50 KB

patch

obsolete

>Subversion Revision: 271356
>diff --git a/Tools/ChangeLog b/Tools/ChangeLog
>index ef54baed088423f69e8016f76a231290f63cba82..df3e8c12c832a39e3c9d982d4da69e285d80a999 100644
>--- a/Tools/ChangeLog
>+++ b/Tools/ChangeLog
>@@ -1,3 +1,28 @@
>+2021-01-22  Alicia Boya GarcÃa  <aboya@igalia.com>
>+
>+        [GTK] run-gtk-tests: Crashing and non-existent tests should not count as a pass
>+        https://bugs.webkit.org/show_bug.cgi?id=220863
>+
>+        Reviewed by NOBODY (OOPS!).
>+
>+        The current implementation doesn't account for test binaries that have
>+        not emitted any subtests. This is the case when a test binary doesn't
>+        exist or it crashes.
>+
>+        Also, in the latter case, the stderr of the crashed processes was not
>+        being outputted to the user, masking a crashing test binary as a
>+        passing test.
>+
>+        This patch fixes both issues: It adds several warning prints when
>+        binaries can't be run or fail to add any subtest, emitting the stderr
>+        generated by the failing or crashing binary.
>+
>+        * glib/api_test_runner.py:
>+        (TestRunner._run_test):
>+        (TestRunner.run_tests):
>+        * glib/glib_test_runner.py:
>+        (GLibTestRunner.run):
>+
> 2021-01-22  Alicia Boya GarcÃa  <aboya@igalia.com>
> 
>         [GTK] run-gtk-tests: Use sys.exit(1) instead of return 1
>diff --git a/Tools/glib/api_test_runner.py b/Tools/glib/api_test_runner.py
>index 94cba57c2a2fcdc7183bcc1645a81fa9820283f2..c5ec12fadac3f8eba015b4e9400c33d87fd43753 100755
>--- a/Tools/glib/api_test_runner.py
>+++ b/Tools/glib/api_test_runner.py
>@@ -277,6 +277,7 @@ class TestRunner(object):
>         if self.is_qt_test(test_program):
>             return self._run_test_qt(test_program)
> 
>+        sys.stderr.write("WARNING: %s doesn't seem to be a supported test program.\n" % test_program)
>         return {}
> 
>     def run_tests(self):
>@@ -303,6 +304,11 @@ class TestRunner(object):
>                 skipped_subtests = self._test_cases_to_skip(test)
>                 number_of_total_tests += len(skipped_subtests if not subtests else set(skipped_subtests).intersection(subtests))
>                 results = self._run_test(test, subtests, skipped_subtests)
>+                if len(results) == 0:
>+                    # No subtests where emitted, either the test binary didn't exist, or we don't know how to run it, or it crashed.
>+                    sys.stderr.write("ERROR: %s failed to run, as it didn't emit any subtests.\n" % test)
>+                    crashed_tests[test] = ["(problem in test executable)"]
>+                    continue
>                 number_of_executed_subtests_for_test = len(results)
>                 if number_of_executed_subtests_for_test > 1:
>                     number_of_executed_tests += number_of_executed_subtests_for_test
>diff --git a/Tools/glib/glib_test_runner.py b/Tools/glib/glib_test_runner.py
>index c8268976c9dd17db60122980dba59f0720cd3d17..b93bc97e157bae6921df0fda133108e104d050fa 100644
>--- a/Tools/glib/glib_test_runner.py
>+++ b/Tools/glib/glib_test_runner.py
>@@ -268,6 +268,12 @@ class GLibTestRunner(object):
>             self._subtest_end(256)
>             need_restart = True
> 
>+        if len(self._results) == 0:
>+            # Normally stderr is checked after a subtest has been parsed. If no subtests have been parsed
>+            # chances are something went wrong with the test executable itself and we should print stderr
>+            # to the user.
>+            sys.stderr.write(self._read_from_stderr(self._stderr_fd))
>+
>         self._stderr_fd = None
> 
>         if need_restart:

Actions: View | Formatted Diff | Diff

Attachments on bug 220863: 418152 | 418421 | 418538 | 418550