224289 2021-04-07 10:05:21 -0700 results.webkit.org should provide API for EWS to check flakiness of tests 2025-02-13 12:03:46 -0800 1 1 1 Unclassified WebKit Tools / Tests WebKit Nightly Build Unspecified Unspecified ASSIGNED https://bugs.webkit.org/show_bug.cgi?id=224434 https://bugs.webkit.org/show_bug.cgi?id=224435 https://bugs.webkit.org/show_bug.cgi?id=204368 https://bugs.webkit.org/show_bug.cgi?id=254148 InRadar P2 Normal --- 1 aakash_jain jbedard aakash_jain ap cgambrell clopez jbedard jenner ryanhaddad webkit-bug-importer oldest_to_newest 1747942 0 aakash_jain 2021-04-07 10:05:21 -0700 results.webkit.org should provide a REST API for EWS to check if the test is passing, consistently failing or flaky. API should accept these parameters: test name(s), commit identifier, test-suite (layout-tests, api-tests etc,), platform (macos, iOS etc.), configuration (debug, release etc.) and any other necessary parameter. API should return whether the test is consistently passing, consistently failing, flaky etc. 1749224 1 aakash_jain 2021-04-12 08:39:47 -0700 This might need discussion about the specifics of the API we might need for EWS, specifically for flakiness information. I think we can tackle the problem in two parts: API for dealing with flaky failures in EWS, API for dealing with consistent failures in EWS. For consistent failures, I filed two specific API requests in Bug 224434 and Bug 224435. 1749247 2 jbedard 2021-04-12 09:23:56 -0700 I think the way that this API should work is that is should provide a "percent likelihood" for each outcome of a given test with a given configuration at a given commit. We will need to toy with the algorithm a bit to figure out what the appropriate way to rank commits surrounding the commit in question is, I'm envisioning a result that looks something like this: { "PASS": 80, "FAIL": 10, "TIMEOUT": 5, "CRASH": 5 } Meaning that given the configuration that the user provided, we would expect that the given test passes 80% of the time, fails 10% of the time, timeout 5% of the time and crashes 5% of the time. From that point, EWS can decide if the pass percentage is high enough to justify failing the build. 1750177 3 webkit-bug-importer 2021-04-14 10:06:17 -0700 <rdar://problem/76651206> 1762563 4 429332 cgambrell 2021-05-21 13:40:33 -0700 Created attachment 429332 Current mock-up of script 429332 2021-05-21 13:40:33 -0700 2021-05-21 13:40:33 -0700 Current mock-up of script results.py text/x-python-script 2400 cgambrell IyEvdXNyL2Jpbi9lbnYgcHl0aG9uMwoKaW1wb3J0IGFyZ3BhcnNlCmltcG9ydCBqc29uCmZyb20g dXJsbGliLnBhcnNlIGltcG9ydCBxdW90ZSwgdXJsZW5jb2RlCmZyb20gdXJsbGliLnJlcXVlc3Qg aW1wb3J0IHVybG9wZW4KCnBhcnNlciA9IGFyZ3BhcnNlLkFyZ3VtZW50UGFyc2VyKGRlc2NyaXB0 aW9uPSdHZXQgdGhlIHBlcmNlbnQgY2hhbmNlIG9mIHBhc3MsIHRpbWVvdXQsIGNyYXNoLCBvciBm YWlsdXJlIG9mIGEgdGVzdCcpCgpwYXJzZXIuYWRkX2FyZ3VtZW50KCd0ZXN0JywgdHlwZT1zdHIs IGhlbHA9J1RoZSB0ZXN0IHRvIHJldHVybiByZXN1bHRzIGZvci4nKQpwYXJzZXIuYWRkX2FyZ3Vt ZW50KCctYScsICctLWFyY2hpdGVjdHVyZScsIHR5cGU9c3RyLCBjaG9pY2VzPVsnYXJtNjQnLCAn eDg2JywgJ3g4Nl82NCddLCBoZWxwPSdUaGUgYXJjaGl0ZWN0dXJlIHRvIHBhc3MgdG8gdGhlIHJl c3VsdHMgZGIgY2FsbC4nKQpwYXJzZXIuYWRkX2FyZ3VtZW50KCctZicsICctLWZsYXZvcicsIHR5 cGU9c3RyLCBjaG9pY2VzPVsnZ3B1cHJvY2VzcycsICd3azEnLCAnd2syJ10sIGhlbHA9J1RoZSBm bGF2b3IgdG8gcGFzcyB0byB0aGUgcmVzdWx0cyBkYiBjYWxsLicpCnBhcnNlci5hZGRfYXJndW1l bnQoJy1sJywgJy0tbGltaXQnLCB0eXBlPWludCwgZGVmYXVsdD0xMDAsIGhlbHA9J051bWJlciBv ZiBidWlsZHMgdG8gcmV0dXJuIHJlc3VsdHMgZm9yLicpCnBhcnNlci5hZGRfYXJndW1lbnQoJy1t JywgJy0tbW9kZWwnLCB0eXBlPXN0ciwgY2hvaWNlcz1bJ2lQaG9uZSA4JywgJ2lQYWQgKDV0aCBn ZW5lcmF0aW9uKScsICdpUGhvbmUgU0UgKDFzdCBnZW5lcmF0aW9uKScsICdNYWNtaW5pOCwxJywg J01hY21pbmk5LDEnXSwgaGVscD0nVGhlIG1vZGVsIHRvIHBhc3MgdG8gdGhlIHJlc3VsdHMgZGIg Y2FsbC4nKQpwYXJzZXIuYWRkX2FyZ3VtZW50KCctcCcsICctLXBsYXRmb3JtJywgdHlwZT1zdHIs IGNob2ljZXM9WydHVEsnLCAnaW9zJywgJ21hYycsICd3aW4nLCAnd2luY2FyaW8nLCAnV1BFJ10s IGhlbHA9J1RoZSBwbGF0Zm9ybSB0byBwYXNzIHRvIHRoZSByZXN1bHRzIGRiIGNhbGwuJykKcGFy c2VyLmFkZF9hcmd1bWVudCgnLXMnLCAnLS1zdHlsZScsIHR5cGU9c3RyLCBjaG9pY2VzPVsnZGVi dWcnLCAncmVsZWFzZSddLCBoZWxwPSdUaGUgc3R5bGUgdG8gcGFzcyB0byB0aGUgcmVzdWx0cyBk YiBjYWxsLicpCgp2b2x1bWUgPSBwYXJzZXIuYWRkX211dHVhbGx5X2V4Y2x1c2l2ZV9ncm91cCgp CnZvbHVtZS5hZGRfYXJndW1lbnQoJy12JywgJy0tdmVyYm9zZScsIGFjdGlvbj0nc3RvcmVfdHJ1 ZScpCnZvbHVtZS5hZGRfYXJndW1lbnQoJy1xJywgJy0tcXVpZXQnLCBhY3Rpb249J3N0b3JlX3Ry dWUnKQoKYXJncyA9IHBhcnNlci5wYXJzZV9hcmdzKCkKCnF1ZXJ5X3N0cmluZyA9IHsKCSdsaW1p dCc6IGFyZ3MubGltaXQKfQoKaWYgYXJncy5hcmNoaXRlY3R1cmU6CglxdWVyeV9zdHJpbmdbJ2Fy Y2hpdGVjdHVyZSddID0gYXJncy5hcmNoaXRlY3R1cmUKaWYgYXJncy5mbGF2b3I6CglxdWVyeV9z dHJpbmdbJ2ZsYXZvciddID0gYXJncy5mbGF2b3IKaWYgYXJncy5wbGF0Zm9ybToKCXF1ZXJ5X3N0 cmluZ1sncGxhdGZvcm0nXSA9IGFyZ3MucGxhdGZvcm0KCndpdGggdXJsb3BlbignaHR0cHM6Ly9y ZXN1bHRzLndlYmtpdC5vcmcvYXBpL3Jlc3VsdHMvbGF5b3V0LXRlc3RzL3t9P3t9Jy5mb3JtYXQo cXVvdGUoYXJncy50ZXN0KSwgdXJsZW5jb2RlKHF1ZXJ5X3N0cmluZykpKSBhcyByZXNwb25zZToK CWRhdGEgPSBqc29uLmxvYWRzKHJlc3BvbnNlLnJlYWQoKSkKCQoJZm9yIGkgaW4gZGF0YToKCQly ZXN1bHRzID0gewoJCQknY29uZmlndXJhdGlvbic6IGlbJ2NvbmZpZ3VyYXRpb24nXSwKCQkJJ3Jl c3VsdHMnOiB7CgkJCQknUEFTUyc6IDAsCgkJCQknRkFJTCc6IDAsCgkJCQknVElNRU9VVCc6IDAs CgkJCQknQ1JBU0gnOiAwCgkJCX0KCQl9CgoJCWlmIGxlbihpWydyZXN1bHRzJ10pID09IDA6CgkJ CWJyZWFrCgoJCXBsYWNlID0gbGVuKGlbJ3Jlc3VsdHMnXSkKCQliYXNlID0gMAoJCWZvciBqIGlu IHJhbmdlKDEsIHBsYWNlICsgMSk6CgkJCWJhc2UgKz0gagoKCQlmb3IgcmVzdWx0IGluIGlbJ3Jl c3VsdHMnXToKCQkJaWYgcmVzdWx0WydhY3R1YWwnXSA9PSAnVEVYVCc6CgkJCQlyZXN1bHRbJ2Fj dHVhbCddID0gJ0ZBSUwnCgkJCXJlc3VsdHNbJ3Jlc3VsdHMnXVtyZXN1bHRbJ2FjdHVhbCddXSAr PSBwbGFjZQoJCQlwbGFjZSAtPSAxCgoJCWZvciByZXN1bHRfdHlwZSBpbiByZXN1bHRzWydyZXN1 bHRzJ106CgkJCXJlc3VsdHNbJ3Jlc3VsdHMnXVtyZXN1bHRfdHlwZV0gPSByZXN1bHRzWydyZXN1 bHRzJ11bcmVzdWx0X3R5cGVdIC8gYmFzZSAqIDEwMAoKCQlwcmludChqc29uLmR1bXBzKHJlc3Vs dHMpKQoK