NEW 205280
Python 3: Multiprocessing doesn't do well when nested 
https://bugs.webkit.org/show_bug.cgi?id=205280
Summary Python 3: Multiprocessing doesn't do well when nested
Jonathan Bedard
Reported 2019-12-16 09:59:43 PST
Not quite clear why. We end up with some weird race conditions, I spent a few hours trying to get to the bottom of it with no luck. I can reproduce the problem except in unit testing, so I'm inclined to defer investigation since run-webkit-tests works fine in Python 3 despite this bug.
Attachments
Add attachment proposed patch, testcase, etc.
Alexey Proskuryakov
Comment 1 2019-12-16 11:06:40 PST
Which start method are we using with Python 3?
Jonathan Bedard
Comment 2 2019-12-16 11:13:04 PST
(In reply to Alexey Proskuryakov from comment #1) > Which start method are we using with Python 3? Not sure what you mean by this. For testing, I actually changed the shebang, but the code to run enable run-webkit-tests in Python3 isn't up for review quite yet, polishing that patch now. I'll be referencing this bug in that patch.
Alexey Proskuryakov
Comment 3 2019-12-16 15:28:22 PST
In Python 3, one can choose the underlying method for how multiprocessing works, it's called "start method". See <https://docs.python.org/3/library/multiprocessing.html>. As this bug is about multiprocessing having some strange behavior on tests with Python 3, it's relevant which of the methods you are using. I'm guessing that it is 'spawn' because that's the default. Since 'fork' was the only one that Python 2 supported, that would be the 1:1 replacement, but of course we shouldn't be using that on macOS.
Jonathan Bedard
Comment 4 2019-12-16 15:41:32 PST
(In reply to Alexey Proskuryakov from comment #3) > In Python 3, one can choose the underlying method for how multiprocessing > works, it's called "start method". See > <https://docs.python.org/3/library/multiprocessing.html>. > > As this bug is about multiprocessing having some strange behavior on tests > with Python 3, it's relevant which of the methods you are using. I'm > guessing that it is 'spawn' because that's the default. Since 'fork' was the > only one that Python 2 supported, that would be the 1:1 replacement, but of > course we shouldn't be using that on macOS. I wasn't aware of this. I haven't modified the start method used, so yes, it should be using spawn. Seems possible that the spawn/fork difference is the root of our problem, but I'm still at a loss as to why I saw a race condition.
Note You need to log in before you can comment on or make changes to this bug.