227715 2021-07-06 11:24:26 -0700 [webkitcorepy] run-webkit-tests may hang with python2 after r271683 2021-07-09 01:11:17 -0700 1 1 1 Unclassified WebKit Tools / Tests WebKit Nightly Build Unspecified Unspecified RESOLVED FIXED https://bugs.webkit.org/show_bug.cgi?id=220547 https://bugs.webkit.org/show_bug.cgi?id=226658 InRadar P2 Normal --- 1 clopez clopez aakash_jain bugs-noreply dewei_zhu jbedard mcatanzaro pnormand pvollan webkit-bug-importer oldest_to_newest 1775001 0 clopez 2021-07-06 11:24:26 -0700 Checking the testing EWS for GTK WK2 layout test I noticed most of them were failing to complete the layout-test step and they were hanging at the end. Example: https://ews-build.webkit-uat.org/#/builders/34/builds/34722 09:54:19.733 3 worker/5 imported/w3c/web-platform-tests/service-workers/service-worker/fetch-cors-xhr.https.html passed 09:54:19.748 3 worker/15 imported/w3c/web-platform-tests/streams/piping/flow-control.any.worker.html passed 09:54:19.778 3 worker/6 imported/w3c/web-platform-tests/streams/transform-streams/lipfuzz.any.worker.html passed 09:54:19.780 3 worker/2 "ruby -I /app/webkit/Websites/bugs.webkit.org/PrettyPatch /app/webkit/Websites/bugs.webkit.org/PrettyPatch/prettify.rb /home/ews/worker/GTK-WK2-Tests-EWS/build/layout-test-results/imported/w3c/web-platform-tests/streams/readable-streams/general.any-diff.txt" took 0.18s 09:54:19.780 3 [42759/54367] imported/w3c/web-platform-tests/streams/readable-streams/general.any.html failed unexpectedly (text diff) 09:54:20.079 3 worker/2 imported/w3c/web-platform-tests/streams/readable-streams/general.any.html failed: 09:54:20.079 3 worker/2 text diff 09:54:20.128 3 Some workers failed to gracefully shut down, but in-flight exception taking precedence command timed out: 1200 seconds without output running ['python3', 'Tools/Scripts/run-webkit-tests', '--no-build', '--no-show-results', '--no-new-test-results', '--clobber-old-results', '--release', '--gtk', '--results-directory', 'layout-test-results', '--debug-rwt-logging', '--exit-after-n-failures', '30', '--skip-failing-tests'], attempting to kill process killed by signal 9 program finished with exit code -1 elapsedTime=1895.925172 The issue happens because a combination of two things: - The GTK workers are still using python2 for running the layout tests. This is because the shebang on the scripts points to "python" that defaults to python2. And even if you run the script with "python3" like in "python3 Tools/Scripts/run-webkit-tests" you actually end running it with python2 because of the flatpak-sdk and how the process are re-executed inside the container. We need to change the shebangs here. - A race condition happens when the layout tests are run with "--exit-after-n-failures N" and more than N unexpected failures happen when RWT is executed with python2. When "--exit-after-n-failures" triggers because more unexpected failures than N happen, then a TestRunInterruptedException() exception is raised from the worker and then on the exit handler for the task-queue the workers are terminated via os.kill(). Then the queue of workers is closed, but that causes a hang with python2 when waiting for the threads to be joined. This issue is not reproducible with python3. A way to reproduce this that worked for me reliable is: # Invalidate all the expected results for the css3 tests (truncate them to 10 chars) $ find LayoutTests/css3 -name \*expected.txt -exec truncate -s10 '{}' \; # Run the tests with python2 $ python2 Tools/Scripts/run-webkit-tests --no-build --release --gtk --debug-rwt-logging --exit-after-n-failures 100 fast css3 1775011 1 432954 clopez 2021-07-06 11:38:04 -0700 Created attachment 432954 Patch 1775015 2 clopez 2021-07-06 11:47:18 -0700 (In reply to Carlos Alberto Lopez Perez from comment #0) > > The issue happens because a combination of two things: > > - The GTK workers are still using python2 for running the layout tests. > This is because the shebang on the scripts points to "python" that defaults > to python2. And even if you run the script with "python3" like in "python3 > Tools/Scripts/run-webkit-tests" you actually end running it with python2 > because of the flatpak-sdk and how the process are re-executed inside the > container. We need to change the shebangs here. Seems running with python2 is still the default everywhere but in the UAT? I got confused because layout tests are running with a python3 prefix on https://ews-build.webkit-uat.org/ but not on https://ews-build.webkit.org/ 1775026 3 aakash_jain 2021-07-06 12:12:16 -0700 (In reply to Carlos Alberto Lopez Perez from comment #2) > Seems running with python2 is still the default everywhere but in the UAT? That's right. Jonathan was using UAT for testing Python 3 change. It has not yet landed. See https://bugs.webkit.org/show_bug.cgi?id=226658 1775139 4 clopez 2021-07-06 16:43:30 -0700 (In reply to Aakash Jain from comment #3) > (In reply to Carlos Alberto Lopez Perez from comment #2) > > Seems running with python2 is still the default everywhere but in the UAT? > That's right. Jonathan was using UAT for testing Python 3 change. It has not > yet landed. See https://bugs.webkit.org/show_bug.cgi?id=226658 I see. Then this issue potentially affects all EWS runners because they run with '--exit-after-n-failures 30'. In the event that more than 30 unexpected failures happen then run-webkit-tests will hang (at least on Linux) 1775646 5 mcatanzaro 2021-07-08 04:24:14 -0700 (In reply to Carlos Alberto Lopez Perez from comment #0) > And even if you run the script with "python3" like in "python3 > Tools/Scripts/run-webkit-tests" you actually end running it with python2 > because of the flatpak-sdk and how the process are re-executed inside the > container. We need to change the shebangs here. Note: in freedesktop-sdk, /usr/bin/python is python3. It's only python2 in our runtime because we build our own python2. I believe that is done specifically for it to be used by run-webkit-tests, but maybe we have other developer scripts that still require python2? Phil would probably know for sure. 1775653 6 pnormand 2021-07-08 04:50:17 -0700 https://bugs.webkit.org/show_bug.cgi?id=221043#c7 1775881 7 clopez 2021-07-08 17:22:20 -0700 Can someone please review this? Now this is causing issues on our Debug bot: See https://build.webkit.org/#/builders/63/builds/1638 15:27:52.105 3 [199/58328] accessibility/adjacent-continuations-cause-assertion-failure.html failed unexpectedly (WebKitWebProcess crashed [pid=11420]) 15:27:53.613 3 worker/0 killing driver 15:27:53.613 3 worker/0 accessibility/adjacent-continuations-cause-assertion-failure.html failed: 15:27:53.613 3 worker/0 WebKitWebProcess crashed [pid=11420] 15:27:53.664 3 Some workers failed to gracefully shut down, but in-flight exception taking precedence command timed out: 1200 seconds without output running ['python', './Tools/Scripts/run-webkit-tests', '--no-build', '--no-show-results', '--no-new-test-results', '--clobber-old-results', '--builder-name', 'GTK-Linux-64-bit-Debug-Tests', '--build-number', '1638', '--buildbot-worker', 'gtk-linux-bot-7', '--buildbot-master', 'build.webkit.org', '--report', 'https://results.webkit.org', '--exit-after-n-crashes-or-timeouts', '50', '--exit-after-n-failures', '500', '--debug', '--gtk', '--results-directory', 'layout-test-results', '--debug-rwt-logging'], attempting to kill process killed by signal 9 program finished with exit code -1 elapsedTime=1635.329821 It seems some commit caused more than 500 failures and the bot should be exiting early, but instead of exiting early is hanging due to this bug. 1775887 8 clopez 2021-07-08 17:41:55 -0700 (In reply to Carlos Alberto Lopez Perez from comment #7) > It seems some commit caused more than 500 failures and the bot should be > exiting early, but instead of exiting early is hanging due to this bug. Actually is crashing and exiting due to more than 50 crashes :\ 1775888 9 433190 clopez 2021-07-08 17:45:27 -0700 Created attachment 433190 Patch Just an edit on the comment 1775938 10 ews-feeder 2021-07-09 01:10:26 -0700 Committed r279775 (239544@main): <https://commits.webkit.org/239544@main> All reviewed patches have been landed. Closing bug and clearing flags on attachment 433190. 1775939 11 webkit-bug-importer 2021-07-09 01:11:17 -0700 <rdar://problem/80364971> 432954 2021-07-06 11:38:04 -0700 2021-07-08 17:45:22 -0700 Patch bug-227715-20210706193803.patch text/plain 1902 clopez U3VidmVyc2lvbiBSZXZpc2lvbjogMjc5NTY2CmRpZmYgLS1naXQgYS9Ub29scy9DaGFuZ2VMb2cg Yi9Ub29scy9DaGFuZ2VMb2cKaW5kZXggOGI0YzM3NTBhMGVjMTdjZDAzOWZjMWU0MDkwZmFlMjlm OGUyZmU3Yy4uOTgzMGRlMWJiMzQ0ZjA1Njc3YTI3NDM1NGQyZjZiMTY0MmZlY2VlMiAxMDA2NDQK LS0tIGEvVG9vbHMvQ2hhbmdlTG9nCisrKyBiL1Rvb2xzL0NoYW5nZUxvZwpAQCAtMSwzICsxLDE4 IEBACisyMDIxLTA3LTA2ICBDYXJsb3MgQWxiZXJ0byBMb3BleiBQZXJleiAgPGNsb3BlekBpZ2Fs aWEuY29tPgorCisgICAgICAgIFt3ZWJraXRjb3JlcHldIHJ1bi13ZWJraXQtdGVzdHMgbWF5IGhh bmcgd2l0aCBweXRob24yIGFmdGVyIHIyNzE2ODMKKyAgICAgICAgaHR0cHM6Ly9idWdzLndlYmtp dC5vcmcvc2hvd19idWcuY2dpP2lkPTIyNzcxNQorCisgICAgICAgIFJldmlld2VkIGJ5IE5PQk9E WSAoT09QUyEpLgorCisgICAgICAgIFdoZW4gYW4gZXhjZXB0aW9uIGlzIHJhaXNlZCBmcm9tIGEg d29ya2VyIGlmIHRoZSB3b3JrZXJzIGFyZSB0ZXJtaW5hdGVkCisgICAgICAgIHZpYSBhIFNJR0tJ TEwgc2lnbmFsIHRoYXQgbGF0ZXIgY2F1c2VzIHRoZSB0YXNrLXF1ZXVlIHRvIGhhbmcgYXQgY2xv c2UoKS4KKyAgICAgICAgVGhlIGlzc3VlIGlzIG5vdCByZXByb2R1Y2libGUgd2l0aCBweXRob24z LiBTbyBrZWVwIHRoZSBzYW1lIGJlaGF2aW91ciBmb3IKKyAgICAgICAgcHl0aG9uMywgYW5kIGZv ciBweXRob24yIGp1c3QgdXNlIHRlcm1pbmF0ZSgpIHRvIHN0b3AgdGhlIHdvcmtlcnMuCisKKyAg ICAgICAgKiBTY3JpcHRzL2xpYnJhcmllcy93ZWJraXRjb3JlcHkvd2Via2l0Y29yZXB5L3Rhc2tf cG9vbC5weToKKyAgICAgICAgKFRhc2tQb29sLl9fZXhpdF9fKToKKwogMjAyMS0wNy0wNSAgUGhp bGlwcGUgTm9ybWFuZCAgPHBub3JtYW5kQGlnYWxpYS5jb20+CiAKICAgICAgICAgW0dUS10gUGFj ayBoZWFkZXIgYW5kIC5wYyBmaWxlcyBpbiB0aGUgYnVpbHQtcHJvZHVjdCBhcmNoaXZlCmRpZmYg LS1naXQgYS9Ub29scy9TY3JpcHRzL2xpYnJhcmllcy93ZWJraXRjb3JlcHkvd2Via2l0Y29yZXB5 L3Rhc2tfcG9vbC5weSBiL1Rvb2xzL1NjcmlwdHMvbGlicmFyaWVzL3dlYmtpdGNvcmVweS93ZWJr aXRjb3JlcHkvdGFza19wb29sLnB5CmluZGV4IDJmNzA2MjFhNzcwMDhlN2FiNDBhMmM1Y2VlZmM4 NzU5ZWYwZTIyYjkuLmFlNDUwMzUwZDVmYWNiYzMyZjQzZTA1YTdmYjc2OWZhNGI2YjNkOWUgMTAw NjQ0Ci0tLSBhL1Rvb2xzL1NjcmlwdHMvbGlicmFyaWVzL3dlYmtpdGNvcmVweS93ZWJraXRjb3Jl cHkvdGFza19wb29sLnB5CisrKyBiL1Rvb2xzL1NjcmlwdHMvbGlicmFyaWVzL3dlYmtpdGNvcmVw eS93ZWJraXRjb3JlcHkvdGFza19wb29sLnB5CkBAIC00NTgsNyArNDU4LDggQEAgY2xhc3MgVGFz a1Bvb2wob2JqZWN0KToKIAogICAgICAgICAgICAgICAgIGlmIHN5cy52ZXJzaW9uX2luZm8gPj0g KDMsIDcpOgogICAgICAgICAgICAgICAgICAgICB3b3JrZXIua2lsbCgpCi0gICAgICAgICAgICAg ICAgZWxpZiBoYXNhdHRyKHNpZ25hbCwgJ1NJR0tJTEwnKToKKyAgICAgICAgICAgICAgICAjIHdl YmtpdC5vcmcvYi8yMjc3MTUgd2l0aCBweXRob24yIGtpbGxpbmcgZGlyZWN0bHkgdGhlIHdvcmtl cnMgY2F1c2VzIHRoZSBxdWV1ZSB0byBoYW5nIG9uIGNsb3NlKCkKKyAgICAgICAgICAgICAgICBl bGlmIGhhc2F0dHIoc2lnbmFsLCAnU0lHS0lMTCcpIGFuZCBzeXMudmVyc2lvbl9pbmZvLm1ham9y ID4gMjoKICAgICAgICAgICAgICAgICAgICAgdHJ5OgogICAgICAgICAgICAgICAgICAgICAgICAg b3Mua2lsbCh3b3JrZXIucGlkLCBzaWduYWwuU0lHS0lMTCkKICAgICAgICAgICAgICAgICAgICAg ZXhjZXB0IE9TRXJyb3IgYXMgZToK 433190 2021-07-08 17:45:27 -0700 2021-07-09 01:10:27 -0700 Patch bug-227715-20210709014523.patch text/plain 1879 clopez U3VidmVyc2lvbiBSZXZpc2lvbjogMjc5NzY5CmRpZmYgLS1naXQgYS9Ub29scy9DaGFuZ2VMb2cg Yi9Ub29scy9DaGFuZ2VMb2cKaW5kZXggNGYxYjNlYWZjNjg4MzhmMjA5MmZmNDJjMDgxNDEwNjYy MmZhZWJmOS4uNTU5MDQ5MGFlNWFkMmU2ODdiOGM3YzQ5MjMyOTZlMzY2N2YxYWRiYiAxMDA2NDQK LS0tIGEvVG9vbHMvQ2hhbmdlTG9nCisrKyBiL1Rvb2xzL0NoYW5nZUxvZwpAQCAtMSwzICsxLDE4 IEBACisyMDIxLTA3LTA2ICBDYXJsb3MgQWxiZXJ0byBMb3BleiBQZXJleiAgPGNsb3BlekBpZ2Fs aWEuY29tPgorCisgICAgICAgIFt3ZWJraXRjb3JlcHldIHJ1bi13ZWJraXQtdGVzdHMgbWF5IGhh bmcgd2l0aCBweXRob24yIGFmdGVyIHIyNzE2ODMKKyAgICAgICAgaHR0cHM6Ly9idWdzLndlYmtp dC5vcmcvc2hvd19idWcuY2dpP2lkPTIyNzcxNQorCisgICAgICAgIFJldmlld2VkIGJ5IE5PQk9E WSAoT09QUyEpLgorCisgICAgICAgIFdoZW4gYW4gZXhjZXB0aW9uIGlzIHJhaXNlZCBmcm9tIGEg d29ya2VyIGlmIHRoZSB3b3JrZXJzIGFyZSB0ZXJtaW5hdGVkCisgICAgICAgIHZpYSBhIFNJR0tJ TEwgc2lnbmFsIHRoYXQgbGF0ZXIgY2F1c2VzIHRoZSB0YXNrLXF1ZXVlIHRvIGhhbmcgYXQgY2xv c2UoKS4KKyAgICAgICAgVGhlIGlzc3VlIGlzIG5vdCByZXByb2R1Y2libGUgd2l0aCBweXRob24z LiBTbyBrZWVwIHRoZSBzYW1lIGJlaGF2aW91ciBmb3IKKyAgICAgICAgcHl0aG9uMywgYW5kIGZv ciBweXRob24yIGp1c3QgdXNlIHRlcm1pbmF0ZSgpIHRvIHN0b3AgdGhlIHdvcmtlcnMuCisKKyAg ICAgICAgKiBTY3JpcHRzL2xpYnJhcmllcy93ZWJraXRjb3JlcHkvd2Via2l0Y29yZXB5L3Rhc2tf cG9vbC5weToKKyAgICAgICAgKFRhc2tQb29sLl9fZXhpdF9fKToKKwogMjAyMS0wNy0wOCAgWWlq aWEgSHVhbmcgIDx5aWppYV9odWFuZ0BhcHBsZS5jb20+CiAKICAgICAgICAgQWRkIFlpamlhIEh1 YW5nIGFzIGEgY29tbWl0dGVyCmRpZmYgLS1naXQgYS9Ub29scy9TY3JpcHRzL2xpYnJhcmllcy93 ZWJraXRjb3JlcHkvd2Via2l0Y29yZXB5L3Rhc2tfcG9vbC5weSBiL1Rvb2xzL1NjcmlwdHMvbGli cmFyaWVzL3dlYmtpdGNvcmVweS93ZWJraXRjb3JlcHkvdGFza19wb29sLnB5CmluZGV4IDJmNzA2 MjFhNzcwMDhlN2FiNDBhMmM1Y2VlZmM4NzU5ZWYwZTIyYjkuLmYyNjczMTkwOGY1ZTViYTE3OTM4 YWUyOTBjN2MwY2QxYWJmNDQxMTUgMTAwNjQ0Ci0tLSBhL1Rvb2xzL1NjcmlwdHMvbGlicmFyaWVz L3dlYmtpdGNvcmVweS93ZWJraXRjb3JlcHkvdGFza19wb29sLnB5CisrKyBiL1Rvb2xzL1Njcmlw dHMvbGlicmFyaWVzL3dlYmtpdGNvcmVweS93ZWJraXRjb3JlcHkvdGFza19wb29sLnB5CkBAIC00 NTgsNyArNDU4LDggQEAgY2xhc3MgVGFza1Bvb2wob2JqZWN0KToKIAogICAgICAgICAgICAgICAg IGlmIHN5cy52ZXJzaW9uX2luZm8gPj0gKDMsIDcpOgogICAgICAgICAgICAgICAgICAgICB3b3Jr ZXIua2lsbCgpCi0gICAgICAgICAgICAgICAgZWxpZiBoYXNhdHRyKHNpZ25hbCwgJ1NJR0tJTEwn KToKKyAgICAgICAgICAgICAgICAjIFdpdGggcHl0aG9uMiBraWxsaW5nIGRpcmVjdGx5IHRoZSB3 b3JrZXJzIGNhdXNlcyB0aGUgcXVldWUgdG8gaGFuZyBvbiBjbG9zZSgpIDxodHRwczovL3dlYmtp dC5vcmcvYi8yMjc3MTU+CisgICAgICAgICAgICAgICAgZWxpZiBoYXNhdHRyKHNpZ25hbCwgJ1NJ R0tJTEwnKSBhbmQgc3lzLnZlcnNpb25faW5mby5tYWpvciA+IDI6CiAgICAgICAgICAgICAgICAg ICAgIHRyeToKICAgICAgICAgICAgICAgICAgICAgICAgIG9zLmtpbGwod29ya2VyLnBpZCwgc2ln bmFsLlNJR0tJTEwpCiAgICAgICAgICAgICAgICAgICAgIGV4Y2VwdCBPU0Vycm9yIGFzIGU6Cg==