140819 2015-01-23 00:43:00 -0800 Fix the false positive build failures on the Windows buildbots 2015-01-27 09:50:54 -0800 1 1 1 Unclassified WebKit New Bugs 528+ (Nightly build) Unspecified Unspecified RESOLVED FIXED https://bugs.webkit.org/show_bug.cgi?id=140845 P2 Normal --- 1 ossy ossy ap bfulgham commit-queue ddkilzer lforschler ossy rniwa roger_fong oldest_to_newest 1063631 0 ossy 2015-01-23 00:43:00 -0800 examples: https://build.webkit.org/builders/Apple%20Win%20Release%20%28Build%29/builds/66798 https://build.webkit.org/builders/Apple%20Win%20Release%20%28Build%29/builds/66807 https://build.webkit.org/builders/Apple%20Win%20Release%20%28Build%29/builds/66832 https://build.webkit.org/builders/Apple%20Win%20Release%20%28Build%29/builds/66849 The problem is that buildbot kills the compile step when there is output since 20 minutes. If a change is too big, the build time takes more than 20 minutes on the Windows bots, but the Visual Studio Express doesn't write anything to stdout during building. The only one fix here is to increase the timeout of this buildstep. 1063633 1 245215 ossy 2015-01-23 00:44:48 -0800 Created attachment 245215 Patch 1063640 2 245217 ossy 2015-01-23 01:09:40 -0800 Created attachment 245217 Patch 1063683 3 245217 bfulgham 2015-01-23 08:48:50 -0800 Comment on attachment 245217 Patch View in context: https://bugs.webkit.org/attachment.cgi?id=245217&action=review I'm in favor of this change, but I think some of the tools people should have final say. It certainly takes longer than 10 minutes to build on Windows from a clean slate; as things currently stand I think we hit this timeout any time we have to rebuild most of WebKit. > Tools/BuildSlaveSupport/build.webkit.org-config/master.cfg:203 > + kwargs['timeout'] = 60 * 60 It certainly can take an hour or so on Windows for a clean build, but I think most of our Mac bots do it in less time. Is there any way to configure this so that Windows has a long timeout and perhaps leave others alone? Or do other bots have a similar timeout issue and we should just change it across the board? 1063686 4 245217 ddkilzer 2015-01-23 08:56:28 -0800 Comment on attachment 245217 Patch View in context: https://bugs.webkit.org/attachment.cgi?id=245217&action=review >> Tools/BuildSlaveSupport/build.webkit.org-config/master.cfg:203 >> + kwargs['timeout'] = 60 * 60 > > It certainly can take an hour or so on Windows for a clean build, but I think most of our Mac bots do it in less time. Is there any way to configure this so that Windows has a long timeout and perhaps leave others alone? > > Or do other bots have a similar timeout issue and we should just change it across the board? If it's only Windows that has this issue, we should only set the timeout to be 2 hours on Windows bots. Can we use self.getProperty('platform') here (or kwargs['platform']) to only set this for Windows bots? 1063687 5 bfulgham 2015-01-23 08:58:30 -0800 (In reply to comment #0) > examples: > https://build.webkit.org/builders/Apple%20Win%20Release%20%28Build%29/builds/ > 66798 > https://build.webkit.org/builders/Apple%20Win%20Release%20%28Build%29/builds/ > 66807 > https://build.webkit.org/builders/Apple%20Win%20Release%20%28Build%29/builds/ > 66832 > https://build.webkit.org/builders/Apple%20Win%20Release%20%28Build%29/builds/ > 66849 > > The problem is that buildbot kills the compile step when there is output > since 20 minutes. > If a change is too big, the build time takes more than 20 minutes on the > Windows bots, > but the Visual Studio Express doesn't write anything to stdout during > building. > > The only one fix here is to increase the timeout of this buildstep. I wonder if there's any way to output to stdout from the build system as another way to avoid this. 1063688 6 ossy 2015-01-23 08:59:55 -0800 (In reply to comment #3) > It certainly can take an hour or so on Windows for a clean build, but I > think most of our Mac bots do it in less time. Is there any way to configure > this so that Windows has a long timeout and perhaps leave others alone? > > Or do other bots have a similar timeout issue and we should just change it > across the board? The default 20 minutes timeout isn't mean that the build should be finished in 20 minutes. The buildmaster will kill the build only if it doesn't produce any output in 20 minutes. It isn't problem for Linux and Mac builders, but Visual Studio Express doesn't produce any output during the build. 1063689 7 ossy 2015-01-23 09:04:02 -0800 (In reply to comment #4) > Comment on attachment 245217 [details] > If it's only Windows that has this issue, we should only set the timeout to > be 2 hours on Windows bots. > > Can we use self.getProperty('platform') here (or kwargs['platform']) to only > set this for Windows bots? Unfortunately it isn't so easy, because properties aren't accessible when the buildstep is instantiated in BuildFactory". But maybe we can pass the platform to CompileWebKit() or instantiate with timeout if platform=="win". Let me try it. 1063694 8 245230 ossy 2015-01-23 09:17:23 -0800 Created attachment 245230 Patch I checked, CompileWebKit(timeout=xxxx) works and platform is passed to BuildFactory.__init__, so we can set timeout for Windows only. 1063710 9 ap 2015-01-23 10:09:06 -0800 That's an amazing find! Can't wait for these failures to be a thing of the past. Two comments: 1. Why are Windows builds so slow? We have decent hardware as far as I know, is there some sort of misconfiguration by any chance? This comment obviously doesn't block reviewing or landing the fix. 2. I think that a better way to fix this would be to actually pipe the logs to output as they come. This way, one could watch progress in real time on a web page, like we do for non-Windows builds. And the bot wouldn't remain stuck for a longer time when an actual freeze occurs. Is that possible to implement with Visual Studio? I think that we should fix it this way if possible. 1063756 10 bfulgham 2015-01-23 13:30:36 -0800 (In reply to comment #9) > That's an amazing find! Can't wait for these failures to be a thing of the > past. > > Two comments: > > 1. Why are Windows builds so slow? We have decent hardware as far as I know, > is there some sort of misconfiguration by any chance? > > This comment obviously doesn't block reviewing or landing the fix. The build itself is no slower than on Mac. The problem is that the way the VS build runs, it does not output any information to stdout; therefore the script thinks the build is hung. I spent about 5 minutes looking for a setting in Visual Studio to output the "Output Window" text to stdout, but didn't find anything. Another option might be to switch to using MSBuild directly, rather than driving it via Visual Studio. This would give us stdout logging like we have using xcodebuild on Mac. The only downside would be having to write an MSBuild input file to control things, but that's not a huge problem. > 2. I think that a better way to fix this would be to actually pipe the logs > to output as they come. This way, one could watch progress in real time on a > web page, like we do for non-Windows builds. And the bot wouldn't remain > stuck for a longer time when an actual freeze occurs. Is that possible to > implement with Visual Studio? > > I think that we should fix it this way if possible. I'll do a little more digging before giving up. Alternatively, we could land this patch now and revise things later if we figure out how to do it. If we leave things as they stand, we get false build failures from time-to-time. With ossy's proposed patch, that wouldn't happen anymore. 1063764 11 ap 2015-01-23 13:49:10 -0800 > The build itself is no slower than on Mac A clean OS X build takes 44 minutes on a Mac mini (see <https://build.webkit.org/builders/Apple%20Yosemite%20Release%20%28Build%29/builds/2387>). I don't know how long each target takes, but even WebCore is likely under 20 minutes. Also, Mac mini is not all that fast. 1063765 12 bfulgham 2015-01-23 13:51:50 -0800 (In reply to comment #11) > > The build itself is no slower than on Mac > > A clean OS X build takes 44 minutes on a Mac mini (see > <https://build.webkit.org/builders/Apple%20Yosemite%20Release%20%28Build%29/ > builds/2387>). > > I don't know how long each target takes, but even WebCore is likely under 20 > minutes. Also, Mac mini is not all that fast. Yes, but you are getting build output during that period. The Visual Studio process that build-webkit is watching produces no output until the entire build (consisting of all projects) has completed. 1063766 13 ap 2015-01-23 13:57:49 -0800 I understand how not getting the output breaks the build. What I'm saying is that it shouldn't be taking that long in the first place. And as previously mentioned, this question doesn't block the patch at all. 1063798 14 245230 bfulgham 2015-01-23 16:04:20 -0800 Comment on attachment 245230 Patch r=me. I'm filing a separate bug to deal with this more permanently. 1063813 15 245230 commit-queue 2015-01-23 16:49:08 -0800 Comment on attachment 245230 Patch Clearing flags on attachment: 245230 Committed r179043: <http://trac.webkit.org/changeset/179043> 1063814 16 commit-queue 2015-01-23 16:49:14 -0800 All reviewed patches have been landed. Closing bug. 1064488 17 ossy 2015-01-27 05:38:45 -0800 Just out of curiosity I checked the clean build time on the bots: - release bot: 27 mins, 32 secs - https://build.webkit.org/builders/Apple%20Win%20Release%20%28Build%29/builds/66907 - debug bot: 28 mins, 6 secs - https://build.webkit.org/builders/Apple%20Win%20Debug%20%28Build%29/builds/85014 1064543 18 ap 2015-01-27 09:50:54 -0800 This is not fast, but reasonable. Is there no stdout output until all the targets are built? We may not need to pump the output in real time if we can make each target dump the results once done. 245215 2015-01-23 00:44:48 -0800 2015-01-23 01:09:33 -0800 Patch bug-140819-20150123094648.patch text/plain 1322 ossy U3VidmVyc2lvbiBSZXZpc2lvbjogMTc4OTg2CmRpZmYgLS1naXQgYS9Ub29scy9DaGFuZ2VMb2cg Yi9Ub29scy9DaGFuZ2VMb2cKaW5kZXggNWM5MjZiMzJhOTRlYTliYTRiNGU5NmIyNjFmZjNiMWVh MDYxZmQ4Zi4uZTljNTEyOTg3ZDA2ZTI0OTQ1ZDZlYTk2MTFjOTdmMzg0ZTE0OWEwNCAxMDA2NDQK LS0tIGEvVG9vbHMvQ2hhbmdlTG9nCisrKyBiL1Rvb2xzL0NoYW5nZUxvZwpAQCAtMSwzICsxLDEz IEBACisyMDE1LTAxLTIzICBDc2FiYSBPc3p0cm9nb27DoWMgIDxvc3N5QHdlYmtpdC5vcmc+CisK KyAgICAgICAgRml4IHRoZSBmYWxzZSBwb3NpdGl2ZSBidWlsZCBmYWlsdXJlcyBvbiB0aGUgV2lu ZG93cyBidWlsZGJvdHMKKyAgICAgICAgaHR0cHM6Ly9idWdzLndlYmtpdC5vcmcvc2hvd19idWcu Y2dpP2lkPTE0MDgxOQorCisgICAgICAgIFJldmlld2VkIGJ5IE5PQk9EWSAoT09QUyEpLgorCisg ICAgICAgICogQnVpbGRTbGF2ZVN1cHBvcnQvYnVpbGQud2Via2l0Lm9yZy1jb25maWcvbWFzdGVy LmNmZzoKKyAgICAgICAgKENvbXBpbGVXZWJLaXQpOiBJbmNyZWFzZSB0aGUgdGltZW91dCB0byBh biBob3VyLgorCiAyMDE1LTAxLTIyICBEYXZpZCBLaWx6ZXIgIDxkZGtpbHplckBhcHBsZS5jb20+ CiAKICAgICAgICAgY29uZmlndXJlLXhjb2RlLWZvci1pb3MtZGV2ZWxvcG1lbnQgbmVlZHMgdG8g aW1wb3J0IHdlYmtpdGRpcnMucG0KZGlmZiAtLWdpdCBhL1Rvb2xzL0J1aWxkU2xhdmVTdXBwb3J0 L2J1aWxkLndlYmtpdC5vcmctY29uZmlnL21hc3Rlci5jZmcgYi9Ub29scy9CdWlsZFNsYXZlU3Vw cG9ydC9idWlsZC53ZWJraXQub3JnLWNvbmZpZy9tYXN0ZXIuY2ZnCmluZGV4IDgyNmZiNDJiYjJj MjEwNTEwMjRhNzA5Zjc2ZDgyZjEyNzc4OTkwOTcuLjA2OGI1ODcxYTBmZWM0ODIzNGQ3OTlhMWIw NDczODM0MGMxMzZhMjEgMTAwNjQ0Ci0tLSBhL1Rvb2xzL0J1aWxkU2xhdmVTdXBwb3J0L2J1aWxk LndlYmtpdC5vcmctY29uZmlnL21hc3Rlci5jZmcKKysrIGIvVG9vbHMvQnVpbGRTbGF2ZVN1cHBv cnQvYnVpbGQud2Via2l0Lm9yZy1jb25maWcvbWFzdGVyLmNmZwpAQCAtMTk4LDYgKzE5OCw3IEBA IGNsYXNzIENvbXBpbGVXZWJLaXQoc2hlbGwuQ29tcGlsZSk6CiAgICAgZGVzY3JpcHRpb24gPSBb ImNvbXBpbGluZyJdCiAgICAgZGVzY3JpcHRpb25Eb25lID0gWyJjb21waWxlZCJdCiAgICAgd2Fy bmluZ1BhdHRlcm4gPSAiLiphcm5pbmc6IC4qIgorICAgIHRpbWVvdXQgPSA2MCAqIDYwCiAKICAg ICBkZWYgc3RhcnQoc2VsZik6CiAgICAgICAgIHBsYXRmb3JtID0gc2VsZi5nZXRQcm9wZXJ0eSgn cGxhdGZvcm0nKQo= 245217 2015-01-23 01:09:40 -0800 2015-01-23 09:17:16 -0800 Patch bug-140819-20150123101140.patch text/plain 1440 ossy U3VidmVyc2lvbiBSZXZpc2lvbjogMTc4OTg2CmRpZmYgLS1naXQgYS9Ub29scy9DaGFuZ2VMb2cg Yi9Ub29scy9DaGFuZ2VMb2cKaW5kZXggNWM5MjZiMzJhOTRlYTliYTRiNGU5NmIyNjFmZjNiMWVh MDYxZmQ4Zi4uZTljNTEyOTg3ZDA2ZTI0OTQ1ZDZlYTk2MTFjOTdmMzg0ZTE0OWEwNCAxMDA2NDQK LS0tIGEvVG9vbHMvQ2hhbmdlTG9nCisrKyBiL1Rvb2xzL0NoYW5nZUxvZwpAQCAtMSwzICsxLDEz IEBACisyMDE1LTAxLTIzICBDc2FiYSBPc3p0cm9nb27DoWMgIDxvc3N5QHdlYmtpdC5vcmc+CisK KyAgICAgICAgRml4IHRoZSBmYWxzZSBwb3NpdGl2ZSBidWlsZCBmYWlsdXJlcyBvbiB0aGUgV2lu ZG93cyBidWlsZGJvdHMKKyAgICAgICAgaHR0cHM6Ly9idWdzLndlYmtpdC5vcmcvc2hvd19idWcu Y2dpP2lkPTE0MDgxOQorCisgICAgICAgIFJldmlld2VkIGJ5IE5PQk9EWSAoT09QUyEpLgorCisg ICAgICAgICogQnVpbGRTbGF2ZVN1cHBvcnQvYnVpbGQud2Via2l0Lm9yZy1jb25maWcvbWFzdGVy LmNmZzoKKyAgICAgICAgKENvbXBpbGVXZWJLaXQpOiBJbmNyZWFzZSB0aGUgdGltZW91dCB0byBh biBob3VyLgorCiAyMDE1LTAxLTIyICBEYXZpZCBLaWx6ZXIgIDxkZGtpbHplckBhcHBsZS5jb20+ CiAKICAgICAgICAgY29uZmlndXJlLXhjb2RlLWZvci1pb3MtZGV2ZWxvcG1lbnQgbmVlZHMgdG8g aW1wb3J0IHdlYmtpdGRpcnMucG0KZGlmZiAtLWdpdCBhL1Rvb2xzL0J1aWxkU2xhdmVTdXBwb3J0 L2J1aWxkLndlYmtpdC5vcmctY29uZmlnL21hc3Rlci5jZmcgYi9Ub29scy9CdWlsZFNsYXZlU3Vw cG9ydC9idWlsZC53ZWJraXQub3JnLWNvbmZpZy9tYXN0ZXIuY2ZnCmluZGV4IDgyNmZiNDJiYjJj MjEwNTEwMjRhNzA5Zjc2ZDgyZjEyNzc4OTkwOTcuLmJhYTFjODc5MjYxZTI4OGRjYjNlODU5YjYx MDY2OWE2ZTQzNzQyOTggMTAwNjQ0Ci0tLSBhL1Rvb2xzL0J1aWxkU2xhdmVTdXBwb3J0L2J1aWxk LndlYmtpdC5vcmctY29uZmlnL21hc3Rlci5jZmcKKysrIGIvVG9vbHMvQnVpbGRTbGF2ZVN1cHBv cnQvYnVpbGQud2Via2l0Lm9yZy1jb25maWcvbWFzdGVyLmNmZwpAQCAtMTk5LDYgKzE5OSwxMCBA QCBjbGFzcyBDb21waWxlV2ViS2l0KHNoZWxsLkNvbXBpbGUpOgogICAgIGRlc2NyaXB0aW9uRG9u ZSA9IFsiY29tcGlsZWQiXQogICAgIHdhcm5pbmdQYXR0ZXJuID0gIi4qYXJuaW5nOiAuKiIKIAor ICAgIGRlZiBfX2luaXRfXyhzZWxmLCAqKmt3YXJncyk6CisgICAgICAgIGt3YXJnc1sndGltZW91 dCddID0gNjAgKiA2MAorICAgICAgICBzaGVsbC5Db21waWxlLl9faW5pdF9fKHNlbGYsICoqa3dh cmdzKQorCiAgICAgZGVmIHN0YXJ0KHNlbGYpOgogICAgICAgICBwbGF0Zm9ybSA9IHNlbGYuZ2V0 UHJvcGVydHkoJ3BsYXRmb3JtJykKICAgICAgICAgYnVpbGRPbmx5ID0gc2VsZi5nZXRQcm9wZXJ0 eSgnYnVpbGRPbmx5JykK 245230 2015-01-23 09:17:23 -0800 2015-01-23 16:49:08 -0800 Patch bug-140819-20150123181915.patch text/plain 1859 ossy U3VidmVyc2lvbiBSZXZpc2lvbjogMTc5MDA2CmRpZmYgLS1naXQgYS9Ub29scy9DaGFuZ2VMb2cg Yi9Ub29scy9DaGFuZ2VMb2cKaW5kZXggNWM5MjZiMzJhOTRlYTliYTRiNGU5NmIyNjFmZjNiMWVh MDYxZmQ4Zi4uM2YzNzZjMzA4NDRlMmI3YjdhM2Y0YWQ1OWZhOTEyOWVmNzAwNGM2OCAxMDA2NDQK LS0tIGEvVG9vbHMvQ2hhbmdlTG9nCisrKyBiL1Rvb2xzL0NoYW5nZUxvZwpAQCAtMSwzICsxLDE2 IEBACisyMDE1LTAxLTIzICBDc2FiYSBPc3p0cm9nb27DoWMgIDxvc3N5QHdlYmtpdC5vcmc+CisK KyAgICAgICAgRml4IHRoZSBmYWxzZSBwb3NpdGl2ZSBidWlsZCBmYWlsdXJlcyBvbiB0aGUgV2lu ZG93cyBidWlsZGJvdHMKKyAgICAgICAgaHR0cHM6Ly9idWdzLndlYmtpdC5vcmcvc2hvd19idWcu Y2dpP2lkPTE0MDgxOQorCisgICAgICAgIFJldmlld2VkIGJ5IE5PQk9EWSAoT09QUyEpLgorCisg ICAgICAgIEluY3JlYXNlIHRoZSBidWlsZCB0aW1lb3V0ICh3aXRob3V0IHByb2R1Y2luZyBvdXRw dXQpIHRvIDIgaG91cnMgZm9yIFdpbmRvd3MgYm90cywKKyAgICAgICAgdGhlIGRlZmF1bHQgMjAg bWludXRlcyBpcyBlbm91Z2ggZm9yIG90aGVycyBzaW5jZSB0aGV5IHByb2R1Y2Ugb3V0cHV0IGR1 cmluZyB0aGUgYnVpbGQuCisKKyAgICAgICAgKiBCdWlsZFNsYXZlU3VwcG9ydC9idWlsZC53ZWJr aXQub3JnLWNvbmZpZy9tYXN0ZXIuY2ZnOgorICAgICAgICAoQnVpbGRGYWN0b3J5Ll9faW5pdF9f KToKKwogMjAxNS0wMS0yMiAgRGF2aWQgS2lsemVyICA8ZGRraWx6ZXJAYXBwbGUuY29tPgogCiAg ICAgICAgIGNvbmZpZ3VyZS14Y29kZS1mb3ItaW9zLWRldmVsb3BtZW50IG5lZWRzIHRvIGltcG9y dCB3ZWJraXRkaXJzLnBtCmRpZmYgLS1naXQgYS9Ub29scy9CdWlsZFNsYXZlU3VwcG9ydC9idWls ZC53ZWJraXQub3JnLWNvbmZpZy9tYXN0ZXIuY2ZnIGIvVG9vbHMvQnVpbGRTbGF2ZVN1cHBvcnQv YnVpbGQud2Via2l0Lm9yZy1jb25maWcvbWFzdGVyLmNmZwppbmRleCA4MjZmYjQyYmIyYzIxMDUx MDI0YTcwOWY3NmQ4MmYxMjc3ODk5MDk3Li4zY2ZlZDg1MzZjMjk3NDc5MmNmZjNmZDI2MzkwNjM3 OTM2YmJkNzU3IDEwMDY0NAotLS0gYS9Ub29scy9CdWlsZFNsYXZlU3VwcG9ydC9idWlsZC53ZWJr aXQub3JnLWNvbmZpZy9tYXN0ZXIuY2ZnCisrKyBiL1Rvb2xzL0J1aWxkU2xhdmVTdXBwb3J0L2J1 aWxkLndlYmtpdC5vcmctY29uZmlnL21hc3Rlci5jZmcKQEAgLTgxMiw3ICs4MTIsMTIgQEAgY2xh c3MgRmFjdG9yeShmYWN0b3J5LkJ1aWxkRmFjdG9yeSk6CiBjbGFzcyBCdWlsZEZhY3RvcnkoRmFj dG9yeSk6CiAgICAgZGVmIF9faW5pdF9fKHNlbGYsIHBsYXRmb3JtLCBjb25maWd1cmF0aW9uLCBh cmNoaXRlY3R1cmVzLCB0cmlnZ2Vycz1Ob25lLCBhZGRpdGlvbmFsQXJndW1lbnRzPU5vbmUsIFNW Tk1pcnJvcj1Ob25lKToKICAgICAgICAgRmFjdG9yeS5fX2luaXRfXyhzZWxmLCBwbGF0Zm9ybSwg Y29uZmlndXJhdGlvbiwgYXJjaGl0ZWN0dXJlcywgVHJ1ZSwgYWRkaXRpb25hbEFyZ3VtZW50cywg U1ZOTWlycm9yKQotICAgICAgICBzZWxmLmFkZFN0ZXAoQ29tcGlsZVdlYktpdCgpKQorCisgICAg ICAgIGlmIHBsYXRmb3JtID09ICJ3aW4iOgorICAgICAgICAgICAgc2VsZi5hZGRTdGVwKENvbXBp bGVXZWJLaXQodGltZW91dD0yKjYwKjYwKSkKKyAgICAgICAgZWxzZToKKyAgICAgICAgICAgIHNl bGYuYWRkU3RlcChDb21waWxlV2ViS2l0KCkpCisKICAgICAgICAgaWYgdHJpZ2dlcnM6CiAgICAg ICAgICAgICBzZWxmLmFkZFN0ZXAoQXJjaGl2ZUJ1aWx0UHJvZHVjdCgpKQogICAgICAgICAgICAg c2VsZi5hZGRTdGVwKFVwbG9hZEJ1aWx0UHJvZHVjdCgpKQo=