Bug 188413

Summary:

Add basic flakiness-analyzer

Product:

WebKit

Reporter:

Jonathan Bedard <jbedard>

Component:

Tools / Tests

Assignee:

Jonathan Bedard <jbedard>

Status:

NEW ---

Severity:

Normal

CC:

aakash_jain, achristensen, ap, dbates, ddkilzer, ews-watchlist, glenn, lforschler, realdawei, ryanhaddad, tsavell

Priority:

Version:

WebKit Nightly Build

Hardware:

Unspecified

OS:

Unspecified

Attachments:

Description	Flags
Patch	none
Patch	ews-watchlist: commit-queue-
Archive of layout-test-results from ews205 for win-future	none

Description Jonathan Bedard 2018-08-08 11:49:27 PDT

This has come in handy enough times over the last 2 months that it's worth committing.

What this script does is read through the stdout of a test run (or runs) and determines which tests were run before some failing test. It then cross-references successful test runs, with the assumption that if a given test preceded a successful run of another test, it is unlikely to be the cause of the failure. We've been able to track down a few flakey test failures using this script, or a predecessor to it.

Comment 1 Jonathan Bedard 2018-08-08 12:10:42 PDT

Created attachment 346782 [details]
Patch

Comment 2 Jonathan Bedard 2018-08-08 17:35:04 PDT

Comment on attachment 346782 [details]
Patch

Need to figure out why these tests failed on the bots but passed locally.

Comment 3 Jonathan Bedard 2018-08-08 17:47:23 PDT

Created attachment 346807 [details]
Patch

Comment 4 EWS Watchlist 2018-08-11 09:51:51 PDT

Comment on attachment 346807 [details]
Patch

Attachment 346807 [details] did not pass win-ews (win):
Output: https://webkit-queues.webkit.org/results/8828955

New failing tests:
http/tests/security/video-poster-cross-origin-crash2.html

Comment 5 EWS Watchlist 2018-08-11 09:52:02 PDT

Created attachment 346963 [details]
Archive of layout-test-results from ews205 for win-future

The attached test failures were seen while running run-webkit-tests on the win-ews.
Bot: ews205  Port: win-future  Platform: CYGWIN_NT-6.1-2.9.0-0.318-5-3-x86_64-64bit

Comment 6 Ryan Haddad 2018-08-30 10:51:45 PDT

Comment on attachment 346807 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=346807&action=review

> Tools/Scripts/webkitpy/flakiness_analyzer/flakiness_analyzer.py:153
> +        index_to_print = int(raw_input('Pick which of the sequences (0-{}) to print: '.format(len(steps_candidates - 1))))

I hit a bug here when trying to test this locally. This should be: .format(len(steps_candidates) - 1))

Comment 7 Alex Christensen 2021-11-01 12:49:58 PDT

Comment on attachment 346807 [details]
Patch

This has been requesting review for more than one year.  If this is still needed, please rebase and re-request review.